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EXECUTIVE  SUMMARY 

The  present  paper  reports  work  on  a  Small  Business  Innovation  Research  Phase  I  project  to 
develop  a  methodology  and  supporting  tool  for  optimizing  the  use  of  multimedia  technology  to 
support  training.  The  motivation  behind  this  project  comes  from  the  recent  trend  to  move 
training  from  traditional  schoolhouses  to  technology  so  that  costs  can  be  saved  and  training  can 
be  made  readily  available  to  sailors  when  needed.  Because  the  development  of  training 
technology  represents  a  huge  investment,  there  is  a  need  for  formal  methods  to  analyze  how  to 
make  best  use  of  training  resources  so  that  training  needs  can  be  met  in  the  most  cost-effective 
manner.  This  includes  supporting  tradeoff  analyses  in  cases  where  sufficient  resources  are  not 
available  to  support  the  training  community’s  “wish  list”  for  technology. 

In  working  with  the  Navy  on  this  project,  we  identified  two  scenarios  in  which  training 
technology  is  developed.  The  first  is  in  developing  technology  for  new  training  requirements 
where  a  comprehensive  analysis  must  include  developing  the  learning  objectives  and  content  as 
well  as,  selecting  the  right  medium  and  developing  the  technology.  The  second  scenario  is 
converting  current  lecture-based  courses  to  technology-based.  The  latter  is  an  immediate  need, 
the  former  an  ongoing,  long  term  need. 

In  order  to  be  responsive  to  the  needs  of  the  Navy  training  community,  we  established  a 
relationship  with  the  Command,  Naval  Education  and  Training  (CNET).  Their  most  pressing 
need  was  course  conversion.  Specifically,  their  concern  was  whether  technology  could  be  used 
to  save  money  over  current  schoolhouse  instruction.  Their  analysis  was  governed  by  a  goal  of 
realizing  sufficient  cost  savings  such  that  these  savings  would  pay  for  the  conversion  process 
within  five  years. 

Based  on  the  two  identified  needs  (course  conversion  and  development  of  technology  for  new 
training  requirements),  our  project  focused  on  two  broad  tasks.  First,  we  developed  a 
methodology  for  translating  training  requirements  into  the  technologies  that  could  meet  those 
requirements.  This  was  supplemented  by  methods  for  estimating  projected  costs  and  benefits 
associated  with  different  technology  options  so  that  resources  could  be  allocated  to  maximize 
training  benefits  at  the  lowest  possible  cost.  Second,  we  developed  a  costing  tool  to  estimate 
costs  associated  with  converting  courses  to  technology-based  media.  This  tool  was  developed  in 
conjunction  with  CNET.  The  tool  was  tested  on  a  course  that  CNET  was  considering  converting 
to  technology-based  delivery.  Costing  data  used  by  the  tool  was  supplied  by  CNET.  The  results 
of  the  tool’s  analyses  on  course  conversion  costs  matched  CNET’s  own. 
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1.0  INTRODUCTION 

There  is  strong  pressure  to  move  training  from  the  schoolhouse  to  the  field.  The  pressures  to 
do  so  are  twofold.  First,  in  the  light  of  declining  Department  of  Defense  budgets,  there  is  a 
desire  to  reduce  training  costs.  Second,  moving  training  to  the  field  offers  greater  local  control 
so  that  training  may  be  delivered  “just  in  time”  and  more  tailored  to  unit  and  individual  sailor 
needs. 

Technology  is  expected  to  play  a  significant  role  in  this  process.  Technology  can  deliver 
training  at  reduced  manpower  costs  and  can  be  disseminated  widely.  Often  technology  can 
deliver  training  more  effectively  than  traditional  methods.  Multimedia  clearly  offers  a 
tremendous  opportunity  to  create  learning  environments  that  are  richer  and  much  more  realistic 
than  traditional  print-based  media.  In  order  to  optimize  the  use  of  training  media  to  promote 
learning,  a  methodology  is  needed  to  conduct  a  cost-benefit  analysis  that  evaluates  the  costs  of 
developing  and  using  different  technologies  against  the  training  benefits  (e.g.,  increased  learning 
or  decreased  time  to  train  to  criteria)  these  technologies  produce  based  on  the  tasks  and  types  of 
knowledge  being  taught. 

We  note  that  there  are  two  major  scenarios  under  which  training  technology  may  be 
developed  or  adopted.  The  first  is  when  existing  schoolhouse-based  courses  that  need  to  be 
converted  to  technology-based  courses  so  that  they  can  be  moved  to  the  field  or  otherwise 
delivered  in  a  more  cost-effective  manner.  According  to  Vice  Admiral  Tracey,  who  is  Chief  of 
Naval  Education  and  Training  (CNET)  and  Director  of  Naval  Training,  a  major  effort  within  the 
Navy  training  community  over  the  next  few  years  will  be  using  technology  to  move  training  from 
the  schoolhouse  to  the  field  (1997).  In  these  cases,  training  objectives  are  already  well  defined 
and  course  content  is  developed. 

The  second  major  scenario  involves  developing  training  systems  for  new  weapons  systems 
and  other  platforms  from  which  sailors  will  be  required  to  train.  In  such  cases,  the  training 
requirements  need  to  be  developed  along  with  the  training  material. 

1.1  Needs  from  the  course  conversion  perspective. 

We  have  discussed  with  Dr.  Jim  Young  of  Naval  Air  Systems  Command  (NAVAIR),  PMA 
205-3  his  view  of  what  the  needs  of  a  multimedia  training  technology  selection  tool  would  be 
from  the  NAVAIR  perspective.  Below  is  our  understanding  of  the  points  he  made  (we 
apologize  if  we  misunderstood  his  points  and  therefore  inaccurately  relay  them  here). 

According  to  Dr.  Young,  NAV AIR’s  immediate  need  is  to  support  CNET  in  its  course 
conversion  process.  He  described  this  process  as  follows.  A  program  manager  at  CNET  needs 
to  determine  whether  a  course  will  be  converted  from  lecture  to  technology-based.  Any  tool 
must  help  with  a  decision  of  either;  keep  as  is,  convert  to  Computer-Aided  Instruction  (CAI)  or 
Interactive  Courseware  (ICW).  The  analysis  here  must  justify  the  conversion  so  it  can  be  pushed 
through  the  acquisition  process.  CNET  stated  a  goal  of  a  return  on  investment  that  breaks  even 
within  five  years  to  justify  the  acquisition.  After  the  conversion  process  is  done,  measures  of 
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effectiveness  are  taken.  The  measures  of  effectiveness  address  the  question  of  whether  CNET 
gained  anything  to  justify  the  conversion. 

The  above  discussion  illustrates  the  importance  of  cost  projections  and  cost  accountability  in 
these  analyses.  It  is  clear  that  CNET  makes  its  case  for  course  conversion  based  on  whether  it 
can  show  cost  savings  by  using  training  technology  sufficient  enough  to  pay  for  the  course 
conversion  within  a  five  year  period.  Measures  of  effectiveness  also  center  on  evaluating 
whether  the  projected  cost  savings  prior  to  the  conversion  actually  occurred. 

1.2  Needs  from  the  new  training  technology  requirements  perspective. 

Often  a  new  system  is  developed  or  some  other  impetus  that  generates  a  training  requirement. 
Unlike  the  immediate  need  for  course  conversion  described  in  section  1.1,  this  is  a  long  term 
requirement.  In  such  cases,  not  only  do  cost  considerations  come  into  play,  but  it  is  also 
important  to  insure  that  the  technology  developed  will  adequately  meet  the  training  requirement 
for  the  intended  trainee  population.  Here,  there  typically  will  not  be  present  an  existing,  proven 
course  that  meets  the  learning  objectives;  therefore,  the  technology  selection  and  development 
process  is  more  complex.  Media  selection  needs  to  be  justified  on  the  basis  of  its  adequacy  for 
the  job.  Also,  as  technology  provides  new  and  additional  media  for  use  in  training,  the  number 
of  options  available  will  also  increase. 

Here,  we  argue  a  more  formal  approach  may  be  required  for  media  selection  than  is  used  in 
the  typical  course  conversion  process.  The  Navy  has  recently  developed  a  Training  Delivery 
Assessment  Model  (TRAD  AM)  that  step  members  of  the  training  community  through  the 
process  of  developing  courses,  selecting  media,  assessing  effectiveness,  etc.  TRAD  AM  is  too 
extensive  to  review  in  its  entirety  here.  In  section  3.1,  we  discuss  our  own  framework  and  the 
parallels  that  it  has  to  the  TRADAM.  In  doing  so,  we  spell  out  features  of  both  frameworks. 

1.3  Summary  of  Phase  I  accomplishments 

There  were  four  notable  accomplishments  on  the  Phase  I  effort.  These  were: 

1 .  Conduct  a  literature  review  to  determine  the  effectiveness  of  training  technologies 

2.  Develop  a  framework  for  analyzing  training  requirements  so  they  can  be  matched  to  training 
technologies  for  which  a  cost-benefit  analysis  can  be  performed 

3.  Develop  a  costing  tool  for  course  conversion  analysis 

4.  Test  the  tool  on  courses  provided  by  CNET 


2.0  A  LITERATURE  REVIEW  OF  THE  EFFECTIVENESS  OF  TRAINING 

TECHNOLOGIES. 

In  order  to  address  the  long  term  need  of  determining  which  technology  to  use  to  meet 
which  training  requirement,  it  was  important  to  determine  whether  there  was  an  empirical  basis 
to  the  claim  that  technology-based  training  could  meet  training  objectives  at  all.  We  conducted  a 
review  of  the  literature  to  make  this  determination.  While  we  found  that  there  was  variability 
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across  studies,  there  was  a  clear  and  consistent  base  of  evidence  that  suggested  that  technology- 
based  instruction  outperforms  traditional  print-based  and  classroom  instruction  in  terms  of 
leading  to  increased  student  learning  or  decreased  learning  time.  This  suggests  that  technology 
can  be  used  to  supplement  or  replace  traditional  means  of  education,  thereby  meeting  training 
objectives  while  moving  training  to  the  sailor  at  cost  savings. 

We  summarize  some  of  the  results  of  our  initial  literature  review,  which  will  form  the  basis  of 
the  general  population  norms  we  will  use  to  project  training  benefits  in  our  framework.  We  note 
that  most  studies  compare  technology-based  instruction  to  Print-Based  Instruction  (PBI).  The 
type  of  dependent  measure  reported  is  mean  improvement  in  test  performance  measured  in 
Standard  Deviations  (SD).  In  other  words,  if  a  result  of  .5  SD  is  reported,  it  means  that  the 
experimental  (technology  group’s)  mean  performance  was  .5  standard  deviations  higher  than  the 
mean  performance  of  the  control  group.  An  improvement  of  .5  SD  is  roughly  equivalent  to 
increasing  the  mean  performance  (50th  percentile)  of  students  in  the  experimental  condition  to 
that  of  students  at  the  69th  percentile  in  the  control  condition. 

In  our  literature  review,  we  found  that  metaanalysis  studies  were  by  far  the  more  useful 
sources  of  information  due  to  the  fact  that  they  compared  different  multimedia  technology  effects 
with  the  more  traditional  classroom  delivery.  The  effects  of  multimedia  varied  between  grade 
and  age  levels.  We  organized  these  into  five  categories: 

•  Effects  of  Multimedia  on  Elementary  and  Intermediate  School  Students 

•  Effects  of  Multimedia  on  High  School  Students 

•  Effects  of  Multimedia  on  Educationally  Disadvantaged  Students  (K-12) 

•  Effects  of  Multimedia  on  College  Students 

•  Effects  of  Multimedia  on  Adult  Education  (e.g.  military  education) 

Several  studies  (Ryan,  1991)  performed  on  the  Achievement  Effects  of  Microcomputer 
Applications  in  Elementary  School  Students,  showed  that  the  average  student  in  the  treatment 
group  exceeded  the  performance  of  62%  of  the  students  who  were  in  the  control  group.  It  was 
also  found  that  students  in  the  treatment  groups  responded  enthusiastically  to  computer 
instruction.  Another  study,  testing  the  effects  of  peer  interaction  on  computer-based  math 
instruction,  found  that  students  leam  more  effectively  in  groups  than  alone  (Hooper,  1 992).  The 
effects  of  peer  interaction  on  achievement  and  efficiency  on  elementary  school  children  were  also 
observed.  The  results  showed  that  both  achievement  and  efficiency  are  highest  for  high-ability 
homogeneously  grouped  students  and  lowest  on  average-ability  homogeneously  grouped 
students.  Bracey  (1993)  studied  the  effectiveness  of  hypermedia  in  elementary  school  education. 
The  study  found  hypermedia  to  be  more  effective  on  a  long  term  basis  rather  than  on  a  short  term 
basis  (study  lasted  three  years).  One  particular  study  done  on  intermediate  school  instruction 
tested  the  effects  of  word  processing  on  student  writing  in  a  high  computer  access  environment. 

In  this  study  it  was  discovered  that  students  experienced  in  using  word  processors  were  able  to 
produce  better  quality  writing  than  subjects  who  used  paper  and  pencil  (Owston,  Murphy,  et  al., 
1991). 
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The  effects  of  multimedia  in  high  school  instruction  focused  mainly  on  two  subjects:  Science 
and  Mathematics.  Computer  Assisted  Interactive  Video  Disk  Instruction  (CAIVDI)  proved  to  be 
effective  in  science  because  it  offers  a  delivery  system  that  permits  students  to  freely  investigate 
topics  (e.g.  chemical  reactions)  without  the  financial  burden  of  paying  for  chemical  supplies. 
Another  finding  was  that  multimedia  can  actually  adapt  to  the  students  learning  style  as  long  as 
the  student  uses  a  well-designed  multimedia  environment  (Hoffer,  Rodke,  et  al.,  1992).  The  use 
of  simulation  software  also  proved  to  have  interesting  results.  A  comparison  of  Computer 
Assisted  Learning  (CAL)  plus  classroom  and  laboratory  instruction  versus  classroom-lab  work 
(alone)  showed  that  the  experimental  group  achieved  a  significantly  higher  mean  score  on 
academic  achievement  post  test  (Lazarowitz,  Huppert,1993).  There  were  no  significant 
differences  within  groups  by  gender  (in  experimental  or  in  control  groups). 

Another  type  of  simulation  software  tested  was  Interactive  Dissection  (IVD)  Simulation.  In  this 
case  IVD  simulation  was  used  in  two  different  ways:  as  a  substitute  for  dissection  and  as  a 
preparation  tool  to  dissection.  IVD  simulation  as  a  substitute  proved  to  be  at  least  as  effective  as 
actual  dissection  in  promoting  student  learning.  The  use  of  IVD  simulation  as  a  preparation  tool 
was  more  effective  to  students  than  no  preparation  at  all.  IVD  was  also  better  than  viewing  a 
videotape  as  preparation  to  dissection  (Kinzie,  Strauss,  et  al.,  1993). 

The  effects  of  embedding  generative  cognitive  strategies  in  science  software  was  tested  on 
10th  grade  level  students.  The  experimental  group  performed  better  on  post  tests  than  the  control 
group.  The  study  showed  that  embedding  cognitive  strategies  into  science  software 
proportionately  benefits  low  verbal  learners  more  than  high  verbal  learners  (Barba,  Merchant, 
1990). 

Distance  learning  has  also  been  an  important  topic  in  the  use  of  multimedia.  A  student 
satellite-delivered  course  testing  the  student  achievement  and  attitude  found  that  indeed  this  form 
of  delivery  can  be  more  effective  than  standard  instruction  (Martin,  Rainey,  1993). 

In  the  instruction  of  mathematics  at  the  high  school  level  the  focus  was  the  influence  of 
problem  solving  software  on  the  attitudes  of  students.  Computer  Augmented  Instruction  (CAI) 
was  the  subject  of  interest  in  this  case.  Students  using  this  type  of  multimedia  achieved  more 
gains  on  a  test  problem  solving  ability  than  students  not  using  CAL  Furthermore,  they  had  more 
positive  attitudes  about  themselves  when  studying  mathematics.  Also,  female  students  showed 
an  interest  in  wanting  to  work  in  math  related  jobs  in  their  futures  (Funkhouser,  Djang,  1993). 

Another  comparison  made  on  high  school  student  instruction  was  the  use  of  Computer  Based 
Instruction  (CBI)  with  Print  Based  Instruction  (PBI).  The  use  of  CBI  showed  significantly  higher 
motivation  and  higher  scores  in  immediate  recall  over  students  learning  from  PBI  (Yang,  1991- 
1992). 

As  we  can  see,  multimedia  has  a  positive  effect  on  students  from  grade  levels  K-12. 

However,  an  interesting  question  is  whether  multimedia  is  sufficiently  diverse  to  be  helpful  and 
have  a  positive  impact  on  other  cases  such  as  educationally  disadvantaged  students.  An 
investigation  of  the  use  of  CBI  for  providing  remediation  to  this  student  population  showed 
significant  increases  in  both  reading  and  math  performance  (such  increases  showed  to  be 


ll 


Technical  Report  97-012 


educationally  meaningful).  Special  education  students  benefited  most  from  CBI  use  at  all  grade 
levels,  and  remedial  and  general  education  students  benefited  least  at  most  grade  levels  (Swan, 
Guerrero,  et  al.,  1990). 

The  effectiveness  of  CBI  at  the  college  level  had  very  favorable  results  towards  the 
achievement  of  students  in  the  classroom.  CBI  raised  student  exam  scores  by  0.26  SD’s  in  the 
average  study.  CBI  effects  were  somewhat  lower  in  the  hard  and  nonlife  sciences  than  in  social 
sciences  and  education.  CBI  also  produced  small  but,  positive  changes  in  college  student 
attitudes  toward  instruction  and  computers.  Furthermore,  it  was  found  that  CBI  substantially 
reduced  the  amount  of  time  needed  for  instruction  at  the  college  level  (Kulik  &  Kulik,  1986). 
Another  multimedia  technology  that  was  investigated  was  CAIVDI.  The  study  focused  on  the 
effects  of  the  use  of  CAIVDI  on  learning  style  preferences,  attitude  and  GPA  on  the  learner. 
There  were  three  conclusions  concerning  learning  style:  a)  Students  who  perceived  CAIVDI  as 
less  creative  desired  more  mobility  while  learning  and  were  less  persistent  overall;  b)  students 
who  were  more  comfortable  with  the  computer  had  less  need  for  mobility  and  were  more 
persistent;  and  c)  students  who  perceived  the  computer  as  functional  had  less  need  for  mobility 
while  learning.  There  was  no  significant  relationship  found  between  learning  style  preferences 
and  learning  achievement.  One  significant  relationship  between  attitudes  toward  CAIVDI  and 
learner  achievement  was  that  the  student  that  was  more  comfortable  with  the  computer  scored 
higher  on  the  posttest.  GPA  also  seemed  to  have  a  correlation  with  attitude  towards  the  use  of 
multimedia.  Students  with  higher  GPA  were  more  motivated  and  responsive  towards  CAIVDI 
(Billings,  Cobb,  1991). 

According  to  analytical  reviews,  multimedia  use  in  adult  education  such  as  interactive 
Videodisc  Instruction  (VDI)  showed  that  overall,  VDI  improved  achievement  by  0.5  SD’s  over 
less  interactive,  more  conventional  approaches  to  instruction.  The  improvement  is  roughly 
equivalent  to  increasing  the  achievement  of  students  at  the  50th  percentile  to  the  69th  percentile. 
In  Military  training,  and  improvement  of  0.38  SD’s  was  observed  across  several  studies  (roughly 
an  increase  from  50th  percentile  to  65th  percentile).  In  higher  education,  an  improvement  of  .69 
SD’s  was  observed  (50th  to  75th  percentile  achievement).  Interactive  VDI  was  more  effective 
when  the  interactive  features  of  the  medium  were  utilized  (Fletcher,  1990).  We  also  found 
analytical  reviews  on  the  effectiveness  of  Computer-Based  Adult  Education.  Studies  showed 
that  Computer  Based  Education  (CBE)  usually  has  positive  effects  on  adult  learners.  CBE  raised 
the  examination  scores  of  such  students  by  0.42  SD’s  on  the  average  study  (a  statistically 
significant  effect).  Again,  it  was  found  that  CBE  substantially  reduced  the  amount  of  time 
needed  for  instruction.  Reliable  conclusions  could  not  be  reached  about  affect  or  long  term 
cognitive  effects  of  CBE  because  of  the  small  number  of  studies  that  investigated  such  effects 
(Kulik,  Kulik,  et  al.,  1986). 

A  few  studies  have  looked  at  time  to  train  as  a  dependent  measure  rather  than  training 
improvement.  This  dependent  measure  is  of  greater  interest  to  training  organizations  such  as 
CNET.  Their  view  is  that  the  Navy  already  does  an  adequate  job  training  its  personnel. 

However,  reducing  training  time  without  sacrificing  training  quality  will  enable  cost  reductions. 
Orlansky  and  String  (1979)  found  that  computer-based  instruction  resulted  in  a  30%  reduction  in 
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time  to  train  over  traditional  methods.  Fletcher  (1991)  also  reports  a  30%  time  to  train  reduction 
for  multimedia  technology  as  well  as  30%-40%  reduction  in  training  costs. 

CNET  and  others  have  expressed  concern  over  the  variability  of  these  results  and  that  such 
variability  may  make  it  difficult  to  make  reliable  predictions  about  how  effective  a  given 
technology  will  be.  We  agree  that  such  variability  is  problematic.  However,  the  consistent 
positive  effects  shown  by  these  studies  suggest  that  one  could  be  reasonably  confident  in  a 
positive  effect,  even  if  the  size  of  that  effect  had  to  be  represented  by  a  range  of  outcomes  rather 
than  a  precise  outcome.  While  it  is  an  empirical  question,  we  believe  that  more  stable  results 
may  be  possible  if  an  organization  collects  data  on  its  own  student  population  for  its  own  courses 
and  technologies  used.  In  section  3.5.1,  we  discuss  the  use  of  Bayesian  updating  techniques  to 
combine  organization-collected  data  with  more  general  data  reported  in  the  literature. 

The  above  results  do  point  to  a  significant  gap  in  the  literature.  The  types  of  studies  reported 
in  the  literature  focus  on  the  use  of  different  types  of  technology  in  training  subject  areas.  There 
is  tremendous  variability  within  a  technology  medium  such  as  Interactive  Videodisk  as  well  as  a 
subject  area  such  as  mathematics.  This  could  also  explain  the  variability  in  the  results  observed. 
The  studies  we  reviewed  did  not  focus  on  the  components  of  the  technology  (e.g.,  video  vs. 
audio)  or  the  type  of  problem  solving  skills  (e.g.,  critical  thinking  vs.  formula  execution)  being 
taught  in  the  subject  area  to  see  whether  there  were  sharper  relationships  between  them. 

The  primary  purpose  of  the  reported  work  was  to  develop  a  framework  to  prescribe  what 
types  of  problem  solving  skills  would  be  best  taught  by  what  types  of  technology.  The  above 
cited  research  addresses  the  general  question  as  to  whether  technology  can  lead  to  improved  or 
quicker  learning  when  used  instead  of  Print-based  Instruction  or  classroom  instruction.  The 
remainder  of  the  report  outlines  a  framework  for  decomposing  training  requirements  into  the 
underlying  knowledge  and  skills  to  be  taught  and  matches  these  to  the  technology  components 
that  are  hypothesized  to  be  best  suited  for  training  these  knowledge  and  skills. 


3.0  DEVELOP  A  METHODOLOGY  FOR  OPTIMIZING  THE  COST-BENEFIT  RATIO 

OF  USING  MULTIMEDIA  IN  TRAINING. 

The  main  activity  in  Phase  I  was  to  begin  the  development  of  a  methodology  for  optimizing 
the  use  of  multimedia  for  training  both  in  terms  of  training  benefit  and  costs.  Since  the  start  of 
the  Phase  I  project,  the  Defense  Standards  Workgroup  has  developed  TRAD  AM  to  serve  as  a 
guide  for  members  of  the  training  community  to  develop  and  deliver  technology-based  training. 
This  development  caused  us  to  refocus  our  efforts  to  be  more  compatible  with  directions  the 
Navy  is  taking  in  these  areas.  Fortunately,  our  framework  and  that  described  in  TRAD  AM  are 
highly  compatible. 

TRADAM  is  a  more  comprehensive  process  than  what  we  developed  in  Phase  I.  Therefore, 
in  demonstrating  how  our  project  is  designed  to  complement  and  enhance  TRADAM,  we  will 
only  focus  on  those  portions  of  the  TRADAM  that  are  relevant  to  our  Phase  I  framework.  We  do 
note,  however,  that  because  of  our  interest  in  supporting  the  PM  A  205-3  stated  requirement  of 
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supporting  CNET’s  course  conversion  efforts,  our  discussions  on  costing  (particularly  in 
describing  the  tool  we  will  develop)  and  the  tradeoff  techniques  are  likely  to  be  more  detailed 
than  what  is  discussed  in  TRAD  AM. 

3.1  Translation  of  course  objectives  into  knowledge  and  skills  requirements  for  training. 

The  first  step  in  the  development  of  a  training  requirement  is  to  develop  a  set  of  course 
objectives.  These  are  typically  driven  by  tasks  that  the  Navy  (or  other  organization)  expects  their 
personnel  to  perform.  Our  methodology  assumes  that  this  step  has  already  been  performed.  This 
is  one  that  is  covered  by  TRAD  AM  but  is  not  part  of  our  framework  as  the  Phase  I  project  does 
not  offer  any  innovations  in  this  area. 

The  next  step  in  the  TRAD  AM  process  is  to  develop  the  knowledge,  skills  and  attitudes 
necessary  to  perform  these  tasks.  The  framework  we  present  for  doing  this  focuses  on 
knowledge  and  skills,  but  not  attitudes.  Given  the  comprehensiveness  of  TRAD  AM,  we  have 
made  a  deliberate  decision  to  focus  on  those  portions  which  we  feel  we  have  mature  frameworks 
and  approaches  to  minimize  risk  and  position  ourselves  for  success  in  the  project. 

Our  framework  for  decomposing  tasks  into  underlying  knowledge  and  skills  is  based  on 
Research  Development  Corporation’s  Integrated  Knowledge  Structure  or  INKS  framework. 

INKS  is  based  on  knowledge  representation  formalisms  found  in  the  cognitive  psychology  and 
cognitive  science  literatures. 

3.1.1  The  INKS  framework. 


Each  task  a  person  may  be  expected  to  perform  has  associated  problem  solving  knowledge.  The 
goal  of  this  step  is  to  link  Navy  tasks  to  that  underlying  knowledge.  Research  indicates  that 
problem  solving  knowledge  is  diverse  and  well  integrated  (Laskey,  Leddo,  and  Bresnick,  1989; 
Leddo  et  al.,  1 990).  People  have  a  variety  of  problem  solving  strategies  at  their  disposal  and  can 
apply  them  as  called  for  by  the  situation.  These  strategies  are  functional  in  nature  and  are  oriented 
toward  the  goals  and  objectives  that  characterize  the  person's  job. 

A  central  theme  in  knowledge  is  its  functional  orientation.  Knowledge  is  centered  around  goals. 
As  a  result,  any  modeling  of  knowledge  and  implications  derived  for  instruction  must  take  goals 
into  account.  In  fact,  research  by  the  Yale  University  Cognitive  Science  Group  (cf.,  Galambos, 
Abelson  and  Black,  1986)  suggests  that  goals  play  a  powerful  role  in  organizing  people's 
knowledge  in  general.  This  point  is  important  because  we  believe  that  much  of  the  high  level 
decision  making  is  more  goal  driven  than  procedural.  While  researchers  such  as  Anderson  (1982) 
have  argued  that  expert  knowledge  is  characterized  by  procedural  knowledge,  Leddo  et  al.  (1990) 
find  that  true  expertise  is  characterized  by  goal  and  causal  knowledge  while  procedural  knowledge 
actually  characterizes  experienced  nonexperts  (who  are  more  advanced  than  novices  but  are  not  true 
experts).  The  research  described  above  focuses  on  the  behaviors  exhibited  by  people.  We  must 
now  look  at  how  to  translate  knowledge  of  these  behaviors  into  knowledge  components  that  could 
be  used  for  identifying  training  technologies  that  could  teach  them. 
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In  the  cognitive  science  and  psychology  literatures,  several  frameworks  have  been  proposed  as 
models  of  knowledge.  These  schemes  tend  to  address  different  types  of  knowledge.  For  example, 
scripts  (Schank,  1982;  Schank  and  Abelson,  1977)  are  used  to  represent  goal  and  planning 
knowledge  that  is  used  in  fairly  routinized  environments.  Scripts  are  generalized  sequences  of  steps 
used  to  achieve  a  goal.  Script-like  schemas  can  also  be  used  to  integrate  bodies  of  knowledge  into  a 
larger  framework. 

Scripts  are  organized  around  goals  people  wish  to  achieve.  Scripts  contain  plans  for  achieving 
those  goals.  In  addition  to  the  planning  information,  scripts  contain  information  such  as  entry 
conditions  (which  state  when  the  script  is  appropriate  to  use),  outcomes  (the  result  of  executing  the 
plan),  actors  (the  people  involved  in  the  plan),  and  props  (physical  objects  that  are  instrumental  in 
plan  execution). 

Knowledge  about  data  patterns  and  how  objects  are  organized  together  can  be  represented  by 
object  frames  (c.f.,  Anderson,  1980;  Minsky,  1975).  Frames  are  very  much  like  scripts  in  that  they 
are  expectancy-driven  organizers  of  knowledge.  We  conceptualize  scripts  as  focusing  more  on  goal 
and  plan-related  knowledge  while  frames  organize  collections  of  objects.  Frames  can  also  be 
distinguished  from  semantic  nets  (cf.,  Quillian,  1966)  which  tend  to  organize  information  about 
individual  concepts  (such  information  being  referred  to  as  the  concepts  “features”)  and  relationships 
between  them  rather  than  collections  of  objects.  For  example,  the  configuration  of  a  computer  that 
is  used  in  training  may  best  be  represented  by  a  frame  since  it  is  a  collection  of  system  components 
and  relationships  among  their  physical  locations  while  a  single  system  component  such  as  a  hard 
disk  may  best  be  represented  by  a  semantic  net  that  describes  its  features  (e.g.,  its  storage  capacity, 
access  speed). 

Knowledge  about  situation-specific  procedures  can  be  represented  by  production  rules  (cf. 
Newell  and  Simon,  1972).  Production  rules  are  expressed  in  the  form  "IF  [antecedent  condition], 
THEN  [consequent  action]",  where  antecedent  conditions  are  situational  conditions  that  determine 
when  procedures  are  to  be  executed  and  consequent  actions  are  the  procedures  executed  under 
those  conditions.  Production  rules  are  useful  in  both  carrying  out  procedures  (e.g.,  "If  this  step  has 
been  completed,  then  do  this  next  step.")  and  also  generating  inferences  (e.g.,  "If  the  following 
problem  features  are  observed,  then  infer  that  this  is  an  [X]  type  of  problem.").  Production  rules 
can  be  distinguished  from  scripts  in  that  scripts  organize  entire  goal-driven  plans,  while  production 
rules  organize  specific  actions.  Scripts  can  be  viewed  as  collections  of  production  rales  much  the 
way  that  frames  can  be  viewed  as  collections  of  semantic  nets. 

Finally,  causal  and  analogical  reasoning  can  be  captured  by  mental  models  (cf.,  de  Kleer  and 
Brown,  1981;  Johnson-Laird,  1983;  Leddo,  Cardie  and  Abelson,  1987).  In  our  framework,  (Leddo, 
Cardie  and  Abelson,  1987),  mental  models  are  viewed  as  encoding  the  causal  rationale  for  why  a 
specific  problem  solving  procedure  is  used.  One  of  the  factors  that  distinguishes  the  way  experts 
solve  problems  from  the  way  nonexperts  do  is  the  former's  heavy  reliance  on  mental  models  and  the 
ability  to  use  them  to  select  an  appropriate  problem  solving  strategy  to  meet  a  set  of  objectives.  We 
argue  that  mental  models  are  key  to  teaching  conceptual  problem  solving  skills,  because  it  is  often 
knowledge  of  goals  and  causal  relationships  that  are  the  key  to  developing  solutions  to  novel  or 
abstract  problems.  We  also  believe  that  mental  models  are,  therefore,  central  to  evaluating 
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situations  and  generating  plans  in  situations  that  are  novel  and  for  which  there  are  no 
preprogrammed  procedures. 

We  have  discussed  five  different  representation  frameworks  (scripts,  object  frames,  semantic 
nets,  production  rules,  and  mental  models)  for  representing  knowledge.  As  we  mentioned  above, 
people  possess  diverse  knowledge  that  is  richer  that  can  be  handled  by  any  single  framework. 
Leddo,  Cardie  and  Abelson  (1987)  developed  an  Integrated  Knowledge  Structure  (INKS) 
framework  that  combines  these  individual  schemes.  In  the  INKS  framework,  scripts  serve  as  the 
general  organizer  of  knowledge,  linking  plans  and  goals  together.  Production  rules  give  situation- 
specific  procedures  to  be  executed  given  conditions  that  arise  during  the  execution  of  a  plan. 

Frames  organize  collections  of  objects  that  are  utilized  in  the  execution  of  plans  while  semantic 
nets  organize  features  of  the  individual  objects  within  a  frame.  Mental  models  provide  the  rationale 
for  why  procedures  are  executed  and  how  they  are  instrumental  in  achieving  objectives. 

The  INKS  framework  has  received  considerable  testing  in  recent  years.  It  has  been  tested 
empirically  as  a  model  of  expert  problem  solving  knowledge  (Leddo  et  al,  1988;  1990);  it  has  been 
used  as  a  model  for  an  AI  problem  solving  system  (Leddo,  Cardie  and  Abelson,  1987),  as  the  basis 
for  developing  knowledge  elicitation  tools  (Leddo  et  al.,  1987;  Leddo  and  Cohen,  1989;  Leddo  et 
al.,  1992;  Leddo  et  al.,  1995),  and  as  the  basis  for  intelligent  tutoring  systems  (Leddo,  Sak  and 
Laskey,  1989,  Leddo,  Laskey  and  Vane,  1992;  Leddo,  Elliot  and  Woolf,  1994;  Leddo,  1995a; 

Leddo  and  Kolodziej,  1996)  and  ITS  authoring  tools  (Leddo,  1995b). 

There  is  a  rough  correspondence  between  the  INKS  framework  and  the  knowledge  and  skills 
described  in  TRAD  AM  (where  “knowledge”  refers  more  to  things  people  can  report  and  “skill” 
refers  more  to  an  ability  to  perform,  as  in  performing  a  task).  Within  TRAD  AM,  there  is  an 
explicit  distinction  between  the  concept  of  “knowledge”  and  “skill”.  Within  INKS,  there  is  not. 
This  is  because  structures  such  as  scripts  and  production  rules  can  serve  both  purposes.  A  script 
can  be  used  to  report  what  knowledge  is  necessary  to  solve  a  problem  and  also  embodies  the 
actions  necessary  to  carry  out  the  plan.  TRAD  AM  refers  to  the  following  types  of  knowledge: 
facts,  rules,  procedures,  discriminations,  and  problem  solving.  While  the  mapping  to  INKS  is 
not  one-to-one,  there  are  strong  similarities.  For  example,  facts  correspond  to  semantic  and 
object  frame  knowledge  depending  upon  whether  the  concept  being  learned  is  a  single  object  or 
configuration  of  them.  Rule  learning  (which  in  TRADAM  refers  to  concepts  that  are  connected 
to  each  other — such  connections  being  reported  by  the  rule)  can  either  map  to  object  frames  if  the 
rule  is  about  how  different  semantic  concepts  group  together  or  to  production  rules  if  the  rule  is 
about  a  set  of  procedures  that  go  together.  Procedures  map  directly  to  production  rules. 
Discriminations  span  all  the  INKS  types.  One  can  make  discriminations  across  semantic  and 
object  frames  based  on  their  features,  across  production  rules  based  on  their  antecedent 
conditions,  across  scripts  based  on  their  goals  and  entry  conditions,  and  across  mental  models 
based  on  their  causal  elements.  Problem  solving  knowledge  maps  to  either  scripts  or  mental 
models  depending  upon  the  level  of  abstraction.  The  table  below  summarizes  these 
correspondences: 


Technical  Report  97-012 


TRAD  AM  Knowledge  INKS  Knowledge  Elements 

Elements 


Facts 

Rules 

Procedures 

Discriminations 

Problem  Solving 


Semantic  Nets,  Object  Frames 
Object  Frames,  Production  Rules 
Production  Rules 
All  Structures 
Scripts,  Mental  Models 


Table  1.  Correspondences  between  TRADAM  and  INKS  Knowledge  Elements 

There  are  a  variety  of  skills  that  are  reported  in  TRADAM.  These  include  perception,  gross 
motor  skills,  readiness,  mechanism,  continuous  movement,  adaptation,  and  origination.  Again, 
there  is  a  mapping  to  the  INKS  framework.  Perception  relates  to  correct  encoding  of  the 
situation.  This  corresponds  to  invoking  the  correct  situation  features  or  entry  conditions  to 
invoke  the  correct  knowledge  structure.  Therefore,  any  INKS  structure  is  applicable.  Similarly, 
readiness  refers  to  a  state  of  being  ready  to  take  action.  This  pertains  to  having  the  appropriate 
INKS  structure  active  that  is  necessary  to  the  situation.  Adaptation  and  origination  refer  to 
modifying  and  creating,  respectively,  new  skills  to  meet  a  new  situation.  Within  INKS,  mental 
models  are  used  by  people  to  create  or  adapt  new  skills  (Leddo  et  al.,  1990).  Gross  motor  skills, 
mechanism  and  continuous  movement  relate  to  scripts  and  production  mles  where  the  level  of 
precision,  movement  continuity,  etc.  are  spelled  out  by  the  structures.  Again,  we  present  the 
similarities  between  the  frameworks  in  the  table  below: 


TRADAM  Skill  Elements  INKS  Skill  Elements 

Perception  All,  based  on  features  or  entry  conditions 

Gross  Motor  Skills  Production  rules,  Scripts 

Readiness  All,  based  on  activation  of  correct 

structure 

Mechanism  Production  rules,  Scripts 

Continuous  Movement  Production  rules,  Scripts  as  specified 

Adaptation  Mental  models 

Origination  Mental  models 

Table  2.  Correspondences  between  TRADAM  and  INKS  Skill  Elements 


3.1.2  Use  of  INKS  to  operationalize  tasks  to  be  trained  into  learning  objectives. 


Both  TRADAM  and  our  framework  call  for  the  training  tasks  to  be  operationalized  into 
learning  objectives.  The  basis  for  both  is  to  use  the  knowledge/skill  framework  (in  TRADAM ’s 
case  it  would  be  knowledge/skill/attitude).  This  can  be  accomplished  by  having  a  trainer  and  a 
Subject  Matter  Expert  (SME)  construct  an  INKS  model  of  the  tasks  to  be  trained.  Ordinarily,  a 
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trained  knowledge  engineer  would  take  part  in  this  process.  In  order  to  facilitate  this  process 
without  the  use  of  a  knowledge  engineer,  tools  could  be  created  to  prompt  trainers  and  SMEs 
create  an  informal  INKS  model  of  the  tasks.  We  use  the  term  “informal”  because  the  rigor 
necessary  to  build  an  expert  system  (which  is  a  typical  application  of  knowledge  engineering) 
does  not  apply  here.  Rather,  the  key  is  to  spell  out  the  knowledge  and  skills  to  be  trained  in 
sufficient  detail  so  that  a  training  model  can  be  developed  around  it.  In  order  to  help  the  process 
of  developing  this  INKS  knowledge,  we  have  developed  a  knowledge  elicitation  technique  called 
Cognitive  Structure  Analysis  (CSA).  This  technique  can  be  used  without  the  aid  of  an  automated 
tool.  We  describe  the  technique  below. 

CSA's  method  of  querying  a  person  is  driven  by  both  structural  and  content  considerations  of  the 
knowledge  structures  in  the  INKS  framework.  For  example,  scripts  not  only  describe  what  events 
will  happen  in  a  context  (the  content  knowledge),  but  also  their  sequence,  their  importance,  etc.  (the 
structural  knowledge).  Therefore,  a  trainer  wishing  to  develop  script-based  knowledge  associated 
with  training  tasks  would  ask  about  the  goals  being  addressed,  the  conditions  under  which  the  script 
is  used,  the  subgoals  to  be  accomplished,  the  steps  to  be  taken  and  the  outcome  that  results  when 
the  script  is  complete.  An  automated  tool  to  assist  with  this  process  would  offer  prompts  to  the  user 
for  each  of  these  components. 

Similarly,  production  rule  knowledge  consists  of  procedures  to  follow  and  the  antecedent 
conditions  that  trigger  those  procedures.  The  system  could  prompt  a  user  to  answer  questions  about 
antecedent  conditions  and  procedures  as  well.  Semantic  knowledge  consists  of  concepts,  their 
features  or  attributes,  instances  (examples)  and  parent  concepts.  Object  frames  are  configurations 
of  these  concepts.  Finally,  mental  models  focus  on  how  causal  elements  (e.g.,  things  that  have 
force)  interact  within  a  context  or  domain  (that  could  be  defined  by  semantic  features).  CSA 
queries  can  be  constructed  for  each  type  of  knowledge.  An  automated  tool  would  present  users 
with  queries  for  each  of  these  and  help  the  trainer  organize  the  knowledge  in  a  tree  or  learning 
objective  hierarchy  as  is  used  in  the  TRAD  AM  process. 

3.1.3  An  example  of  this  process. 

In  order  to  demonstrate  our  methodology  in  action,  we  will  use  a  running  example.  The 
running  example  is  designed  to  be  illustrative  rather  than  complete  or  even  accurate  in  terms  of 
the  content  described.  We  use  as  an  example  that  the  trainee  will  be  able  to  identify  an  enemy 
aircraft  as  friend  or  foe.  The  first  step  is  to  develop  an  INKS  of  this. 

The  semantic  knowledge  that  pertains  to  this  may  be  the  cues  that  one  has  to  make  this 
determination.  For  illustration  purposes  only,  we  will  assume  that  the  cues  are: 

a)  whether  the  aircraft  is  firing  at  you; 

b)  whether  his  Identification  Friend  or  Foe  (IFF)  transponder  signals  friendly; 

c)  what  altitude  the  aircraft  is  flying  at; 

d)  what  speed  the  aircraft  is  flying  at; 

e)  whether  the  aircraft  is  flying  through  a  safety  corridor,  where  each  of  these  cues 
may  be  represented  semantically  as  an  attribute  of  the  aircraft  to  be  identified. 
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Procedures  may  relate  to  inference  rules  that  a  sailor  might  use  to  make  the  IFF  determination. 
These  may  include: 

a)  If  the  aircraft  is  firing  at  you,  treat  as  enemy. 

b)  If  the  aircraft  is  not  firing  at  you  and  its  IFF  transponder  signals  friendly,  treat 
as  friendly. 

c)  If  the  aircraft  is  not  firing  at  you  and  its  IFF  transponder  has  no  signal,  then  use 
a  two  out  of  three  rule  for  the  remaining  cues  to  determine  whether  to  treat  aircraft 
as  friend  or  enemy  (assuming  that  there  is  a  correspondence  between  altitude, 
speed  and  safety  corridor  and  what  friendly  vs.  enemy  planes  would  do). 

Schema  may  include  different  battle  sequences  that  may  help  make  a  determination  on  how  to 
treat  the  remaining  cues.  For  example,  aircraft  tend  to  fly  at  low  altitudes  as  they  make  a 
bombing  run  and  higher  altitudes  en  route  and  on  the  way  back  to  avoid  enemy  air  defense. 
Therefore,  if  the  sailor  is  near  a  strategic  target  and  an  aircraft  is  approaching  at  low  altitude,  it 
may  match  the  cue  of  an  enemy  aircraft  making  a  bombing  run. 

While  we  are  using  this  particular  example,  we  wish  to  point  out  that  not  all  learning 
objectives  will  require  every  type  of  INKS  knowledge  and  skills. 

3.2  Translation  of  knowledge  and  skill  requirements  into  sensory  stimulus  requirements 

Once  the  operationalization  of  the  learning  objectives  into  an  INKS-based  format  is 
completed,  the  INKS-operationalized  learning  objectives  are  translated  into  sensory  stimulus 
requirements.  This  step  is  also  included  in  the  TRADAM  process.  Our  process  is  less 
comprehensive.  The  goal  of  this  step  is  to  form  the  basis  of  determining  what  training 
environments  and  experiences  will  be  necessary  to  train  the  appropriate  knowledge  and  skills. 

In  our  framework,  the  judgment  of  what  sensory  stimulus  requirements  are  necessary  to  train 
particular  knowledge  or  skills  must  be  made  by  the  trainer  or  subject  matter  expert.  It  is 
impractical  to  try  to  develop  a  generalized  tool  that  is  able  to  make  this  translation  itself. 
However,  one  could  provide  a  template  and  set  of  dimensions  that  are  useful  to  categorize  the 
sensory  stimuli.  We  illustrate  our  approach  with  three  dimensions:  sensory  modality  (e.g., 
visual,  auditory,  tactile);  fidelity;  and  configuration  (e.g.,  is  there  spatial  grouping  of  stimuli, 
temporal  sequencing  or  are  stimuli  presented  independently  of  each  other  (which  we  define  as 
“no  configuration”)?). 

The  trainer  would  then  go  through  each  INKS  component  and  make  these  determinations. 

The  table  below  illustrates  this  process  for  the  example  above  (again  keeping  in  mind  that  this  is 
for  illustration  purposes  only  and  the  example  may  not  be  factually  accurate). 
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Cue 

Modality 

Fidelity 

Configuration 

Firing 

Visual 

Low 

Spatial  Grouping 

IFF 

Visual 

Low 

N/A 

Transponder 

Altitude 

Visual 

Low 

Spatial  Grouping 

Speed 

Visual 

Low 

N/A 

Safety  Corridor 

Visual 

High 

Spatial  Grouping  and  Temporal 

Sequencing 


Table  3.  Sensory  Stimulus  Requirements  for  Cues  to  be  Taught 
3.3  Development  of  a  training  strategy 

The  next  step  in  our  methodology  is  to  develop  a  training  strategy  to  meet  the  learning 
objectives  given  the  types  of  sensory  experiences  required.  We  also  take  into  account  trainee 
population  characteristics  as  we  will  explain  shortly.  We  consider  this  part  of  our  model  to 
represent  an  enhancement  to  the  TRADAM  process.  TRAD  AM  does  describe  the  different  types 
of  training  environments  and  the  need  to  take  trainee  characteristics  into  account.  However, 
TRADAM  is  not  formulaic  in  how  this  process  should  occur. 

We  do  confess  that  our  approach,  while  based  on  both  cognitive  theory  and  some  empirical 
findings  is  still  largely  theoretic.  We  recognize  that  this  is  likely  to  be  the  weak  link  in  our  chain 
and  the  one  with  the  largest  risk  factor.  However,  since  our  process  is  designed  to  be  modular, 
this  step  can  be  omitted  or  a  trainer  can  arrive  at  a  training  strategy  based  on  experience  and 
judgment.  Therefore,  the  fact  that  our  hypotheses  about  how  to  accomplish  this  step  may  be 
insufficiently  supported  to  instill  confidence  in  a  user  following  our  recommendations  is  not  fatal 
to  the  overall  methodology  described  here  in  section  3. 

The  goal  of  this  step  is  to  arrive  at  a  training  strategy  that  specifies  how  much  of  different 
types  of  training  should  be  provided  to  trainees  to  have  them  master  the  learning  objectives.  This 
then  will  be  used  to  specify  what  technologies  will  be  used.  We  have  three  dimensions  along 
which  the  training  strategy  will  be  developed.  These  are: 

1 .  The  training  environment  (e.g.,  type,  fidelity); 

2.  the  degree  of  instructor  support  (human  or  artificial)  vs.  self-direction; 

3.  whether  training  is  administered  individually  or  as  a  team. 

Training  environment  types  include  such  things  as  live  practice,  simulator-based  practice 
(constructive  or  virtual),  text-based  instruction,  lecture,  etc.  Fidelity  relates  to  how  close  the 
training  environment  must  be  to  the  actual  experience  in  order  to  achieve  the  desired  level  of 
training. 
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We  use  the  sensory  stimulus  requirements  developed  in  the  section  above  to  make 
determinations  regarding  the  training  environment.  Degree  of  instructor  support  required  is  a 
function  of  both  the  task  to  be  trained  and  the  characteristics  of  the  trainees  to  be  taught  the  task. 
Whether  training  is  administered  individually  or  as  a  team  is  a  function  of  the  task  (e.g.,  is  it 
inherently  a  team  exercise  or  an  individual  one?).  While  training  for  tasks  performed  by 
individuals  can  often  be  delivered  simultaneously  to  a  group  of  people,  the  technology  itself 
would  only  need  to  provide  instructor  support  at  the  individual  level  not  the  team  level.  On  the 
other  hand,  training  for  tasks  performed  by  teams  would  require  instructor  support  at  the  team 
level. 

Because  trainee  population  characteristics  will  impact  the  ideal  learning  environment,  we 
outline  three  relevant  dimensions  here: 

1.  Aptitude 

2.  Experience 

3.  Learning  style 

In  general,  aptitude  has  been  found  to  affect  training  environment  and  amount  of  instructor 
support  required.  Low  aptitude  students  benefit  from  higher  fidelity  environments  and  require 
greater  instructor  support  than  high  aptitude  students  that  seem  to  leam  on  their  own  and 
regardless  of  how  information  is  presented. 

Experience  has  two  effects  on  the  training  strategy.  First,  it  determines  what  learning 
objectives  actually  need  to  be  trained.  If  trainees  have  the  required  skills,  then  the  training 
strategy  can  omit  them.  For  this  reason,  the  TRAD  AM  process  actually  has  this  step  before  the 
analysis  of  sensory  stimulus  requirements  (and  could  be  performed  before  in  our  methodology  if 
desired).  Second,  highly  experienced  students  often  show  similar  effects  with  regard  to  fidelity 
and  self-directed  instruction  as  high  aptitude  students  (probably  the  two  are  typically  correlated 
in  practice),  while  inexperienced  students  typically  show  similar  effects  as  low  aptitude  students. 

A  person’s  learning  style  is  typically  characterized  by  the  factors  in  which  he  learns  best.  The 
most  common  dimension  along  which  learning  style  is  operationalized  is  sensory  modality  by 
which  subject  matter  is  presented.  In  cases  where  the  same  learning  objective  can  be  taught 
using  multiple  sensory  modalities,  then  the  trainer  can  opt  for  the  one  that  matches  trainee 
learning  style  or  give  the  trainee  the  option  to  select  which  mode  he/she  will  leam  by.  Our 
assumption  is  that  the  three  trainee  population  characteristics  that  we  have  described  would  be 
input  by  the  trainer,  if  such  information  is  available.  We  understand  from  our  work  with  CNET 
that  the  Navy  gives  its  recruits  batteries  of  tests  that  measure  a  variety  of  the  recruits’ 
characteristics.  Certainly,  if  not  provided  by  other  sources  of  information,  trainee  experience 
levels  can  be  estimated  from  the  level  of  the  course  and  the  required  prerequisites. 

In  developing  a  training  strategy,  two  approaches  are  possible:  The  first,  is  to  assume  a  mix 
of  training  environments  (e.g.,  a  portion  of  a  course  will  be  converted  to  technology-based).  In 
this  case,  the  trainer  can  try  to  optimize  the  mix  to  produce  the  best  training  benefit  or  the  lowest 


21 


Technical  Report  97-012 


cost.  The  second  approach,  is  to  assume  that  only  one  training  environment  will  be  used.  In  this 
case,  the  trainer  must  look  for  the  minimum  environment  that  meets  all  the  learning  objectives. 

We  now  present  some  illustrative  guidelines  for  mapping  sensory  stimulus  requirements  to 
training  environments.  We  present  a  continuum  of  nonclassroom-based  instructional 
environments.  Each  are  intended  to  represent  a  higher  level  of  fidelity  and  can  accommodate  the 
ones  that  precede  them.  The  environments  (while  illustrative  and  not  exhaustive)  are: 

1.  Text 

2.  Graphics,  audio,  haptic 

3.  Constructive  simulation 

4.  Non-immersive  virtual  simulation,  video 

5.  Immersive  simulation 

6.  Live  practice 

Given  these  environments,  we  present  a  hypothesized  table  of  what  the  minimum  training 
environment  would  be  for  a  learning  objective  of  a  specified  sensory  modality,  fidelity  level,  and 
configuration. 


Low  Fidelity  Sensory  Stimulus  Requirement 


Sensory 

No 

Spatial 

Temporal 

Modality 

Configuration 

Configuration 

Configuration 

Sensory 
Independent 
(i.e.,  can  be 
presented  via 
any  sensory 
modality) 

All  Environments 

N/A 

All  Environments 

Visual 

Graphic 

Graphic 

Graphics  Sequence 

Auditory 

Audio 

Stereo 

Audio  Sequence 

Tactile  (i.e., 
learning  by 
touch) 

Haptic 

Multi-haptic 

Haptic 

Kinesthetic 
(i.e.,  learning 
by  feedback  of 
bodily 
position) 

Immersive 

Simulation 

Immersive 

Simulation 

Immersive  Simulation 

Table  4.  Minimum  Training  Environment  for  Low  Fidelity  Cues  based  on  Sensory 

Modality  and  Configuration 
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High  Fidelity  Sensory  Stimulus  Requirement 


Sensory 

Modality 

Independent 

Visual 

Auditory 

Tactile 

Kinesthetic 


No 

Configuration 

All 

Environments 

Graphic 

Audio 

Haptic 

Live  Exercise 


Spatial 

Configuration 

N/A 

3-D  Graphic 
Stereo 
Multi-haptic 
Live  Exercise 


Temporal 

Configuration 

All  Environments 

Simulation,  Video 
Audio  Sequence 
Haptic 

Live  Exercise 


Table  5.  Minimum  Training  Environment  for  High  Fidelity  Cues  based  on  Sensory 

Modality  and  Configuration 

We  wish  to  make  some  comments  about  the  preceding  tables: 

1  .  First,  in  cases  where  the  same  environment  appears  in  both  low  and  high  fidelity  cells, 
the  fidelity  of  the  environment  would  be  affected  even  if  it  is  the  same.  For  example,  a  single 
auditory  stimulus  may  still  be  presented  by  audio  regardless  of  whether  the  sensory  stimulus 
requirement  is  high  or  low  fidelity,  but  the  fidelity  of  the  audio  would  need  to  correspond  to  the 
fidelity  requirements  of  the  stimulus  (i.e.,  a  high  fidelity  sensory  stimulus  requirement  would 
require  high  fidelity  audio  whereas  a  low  fidelity  sensory  stimulus  requirement  would  require 
low  fidelity  audio). 

2.  Second,  the  minimum  environment  requirements  may  change  as  technology  changes. 
For  example,  we  know  of  no  immersive  environments  that  provide  high  fidelity  kinesthetic 
feedback.  Therefore,  we  cite  live  exercise  as  the  minimum  configuration  for  those  learning 
objectives  that  require  high  fidelity  kinesthetic  feedback.  This  technology  may  change  in  the 
future  and  the  minimum  training  environment  required  would  also  change. 

3.  Third,  there  will  be  many  cases  in  which  the  learning  objective  will  have  multiple 
sensory  modalities.  Each  of  which  may  require  a  different  minimum  training  configuration.  In 
cases  where  multiple  training  environments  are  possible,  this  analysis  would  specify  what  mix  of 
training  environments  are  needed.  We  stated  earlier  that  the  other  possibility  was  when  only  one 
form  of  training  environment  will  be  used.  In  such  cases,  the  minimum  training  configuration 
will  be  the  highest  training  configuration  that  contains  all  of  the  sensory  stimulus  requirements. 

3.3.1  Selecting  training  environments  for  the  running  example  of  IFF. 

We  now  apply  above  analysis  to  our  previous  running  example  of  training  the  IFF  task.  We 
repeat  the  previously  shown  table  of  the  sensory  stimulus  requirements  for  the  training  of 
different  IFF  cues: 
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Cue 

Modality 

Fidelity 

Configuration 

Firing 

Visual 

Low 

Spatial  Grouping 

IFF 

Visual 

Low 

N/A 

Transponder 

Altitude 

Visual 

Low 

Spatial  Grouping 

Speed 

Visual 

Low 

N/A 

Safety  Corridor 

Visual 

High 

Spatial  Grouping  and  Temporal 

Sequencing 


Table  6.  Sensory  Stimulus  Requirements  for  Cues  to  be  Taught 

Based  on  our  hypothesized  match  of  sensory  stimulus  requirements  to  training  environments, 
we  come  up  with  the  minimum  training  environment  required  to  teach  the  use  of  each  cue: 


Cue 

Firing 

IFF  Transponder 

Altitude 

Speed 

Safety  Corridor 


Minimum  Training  Environment 

Graphic 

Graphic 

Graphic 

Graphic 

Virtual  Simulation  or  Video 


Table  7.  Minimum  Training  Environment  for  Cues  to  be  Taught 

As  seen  from  the  above  table,  much  of  the  task  can  be  trained  using  simple  graphics.  The  cue 
of  safety  corridor  requires  a  virtual  simulation  or  video.  Therefore,  if  multiple  training 
environments  are  allowed,  then  a  combination  of  graphics  and  simulation/video  are  possible  and 
these  could  be  constructed  based  on  cost  considerations.  If  only  one  format  is  allowed,  then  it 
would  have  to  be  virtual  simulation  or  video  to  encompass  the  entire  learning  objective. 

3.4  Development  of  technology  requirements. 

Once  the  learning  environment  is  established,  the  next  step  is  to  determine  what  technology 
will  be  used  to  deliver  the  training.  Here,  multiple  technologies  are  potential  options  as  long  as 
they  deliver  the  capabilities  required  (e.g.,  CBT  and  the  Internet  can  deliver  graphics).  We 
recommend  that  in  cases  where  the  trainer  has  the  option  to  pick  the  medium/technology  that 
multiple  options  are  examined  so  that  when  costing  is  done  and/or  tradeoffs  need  to  be  made, 
there  are  multiple  options  to  choose  from  to  try  to  meet  all  the  constraints.  There  may  be  cases 
where  multiple  media  are  selected  as  each  might  cost-effectively  meet  a  portion  of  the  training 
requirement.  In  cases  where  the  medium  or  technology  has  been  selected,  then  the  training 
environment  analysis  will  specify  how  to  construct  the  technology,  which  in  turn  may  help  cost  it 
out. 


We  adopt  the  general  approach  used  by  TRAD  AM  in  conceptualizing  categories  of  training 
technology.  These  include  such  things  as  Interactive  Courseware  (ICW),  Computer-Based 
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Training  (CBT),  Interactive  Videodisk  (IVD),  Internet,  Videoconferencing.  For  each  technology, 
it  is  important  to  develop  a  matrix  of  what  capabilities  it  has.  These  capabilities  match  both  the 
training  environment  dimensions  presented  in  the  previous  section  (e.g.,  graphics,  simulation, 
etc.)  as  well  as  the  dimensions  of  instructor  vs.  self-guided  and  individual  vs.  group  that  is 
related  to  trainee  population  and  task  characteristics,  respectively. 

We  are  cognizant  of  the  fact  that  training  technologies  themselves  evolve  with  a  general  trend 
of  increasing  capability.  For  example,  the  Internet  can  present  graphics  and  text  and  through 
recent  developments,  real  time  audio.  Full  motion  video  and  virtual  simulation  are  problematic, 
but  with  future  advances  in  bandwidth,  these  may  be  possible.  Therefore,  a  commercial 
implementation  of  our  framework  should  include  a  database  of  evolving  technology  capabilities 
so  that  updates  to  the  inputs  used  in  our  framework  regarding  available  technologies  can  be 
made.  For  example,  as  new  video  cards  are  introduced,  PC’s  are  approaching  the  graphics  power 
of  high  end  graphics  machines,  but  at  substantially  lower  costs.  These  advances  may  have  strong 
effects  on  what  technologies  are  selected  for  training. 

In  the  table  below,  we  present  some  sample  media  and  their  current  capabilities. 


Technology 

Maximum  Training 

Instructor 

Individual  or 

Environment 

support 

Team 

Supported 

Training 

Print 

Graphics 

Low 

Individual 

CBT 

Video 

Medium 

Individual 

Distributed 

Virtual  Simulation 

Low 

Team 

Interactive 

Simulation 

IVD 

Video 

Low 

Individual 

Video¬ 

Video 

High 

Both 

teleconferencing 

Internet 

Graphics/Audio 

High 

Both 

Table  8.  Current  Capabilities  of  Sample  Media 

As  we  noted  above,  these  dimensions  can  change  as  technology  changes.  For  example, 
intelligent  tutoring  system  technology  (cf.,  Greer,  1995;  Leddo  and  Kolodziej,  1997)  increases 
the  power  of  CBT  and  DIS  instructor  support  (i.e.,  the  degree  to  which  the  technology  can  act  as 
a  surrogate  instructor).  As  bandwidth  width  increases,  the  maximum  training  environment 
supported  by  the  Internet  will  increase. 

3.4. 1  Selecting  training  technology  for  IFF  task. 

We  can  apply  the  above  technology  capabilities  to  the  IFF  task.  IFF  is  a  task  performed  by  an 
individual.  Depending  upon  the  experience  and  aptitude  of  the  trainee,  it  may  require  low 
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instructor  support.  For  the  purposes  of  this  example  we  will  assume  this  to  be  the  case.  The 
table  below  shows  the  technologies  that  would  support  training  the  IFF  task. 


Cue 


Training  Technologies 


Firing 

IFF  Transponder 

Altitude 

Speed 

Safety  Corridor 


All 

All 

All 

All 

CBT,  IVD,  Videoteleconferencing 


Table  9.  Technologies  That  Could  Support  Training  Each  Cue 


3.5  Projection  of  costs  and  training  benefits  for  each  technology  option 

3.5.1  Projection  of  training  benefits. 

Training  benefits  can  be  measured  in  two  ways:  Increased  learning,  and  time  to  train  to  a 
criterion.  Ultimately,  the  computation  of  training  benefit  is  an  empirical  determination  that  is 
made  after  the  training  is  completed  and  the  outcome  is  assessed.  However,  since  the  present 
project  focuses  on  supporting  decision  making,  projections  of  benefits  should  be  made  before 
training  is  done  or  the  technology  developed.  In  such  cases,  base  rate  data  provides  the  basis  for 
this  projection.  There  are  two  sources  of  base  rate  data.  The  first,  is  to  use  data  taken  from  the 
general  population  based  on  subject  matter  trained,  technology  used,  and  student  population 
characteristics.  The  second,  is  to  use  the  same  data  from  the  training  organization.  There  is  a 
tradeoff  here. 

We  have  reviewed  the  training  technology  literature  and  have  found  tremendous  variability 
across  studies  that  have  been  reported  in  terms  of  the  benefits  of  the  training  technology. 
Therefore,  we  believe  that  the  most  reliable  data  will  come  from  an  organization’s  own  practices 
and  student  populations.  However,  there  tends  to  be  less  of  this  data.  We  recommend  a  hybrid 
solution  that  uses  both  general  population  base  rate  data  and  data  collected  from  the  specific 
training  organization.  In  using  general  population  base  rate  data,  an  organization  should  look  for 
data  that  match,  as  closely  as  possible,  their  student  population  characteristics,  subject  matter, 
and  technology  used.  These  data  form  the  initial  projection  of  training  benefit  for  using 
technology  to  train  the  organization’s  student  population.  Unfortunately,  most  empirical  studies 
have  focused  on  increased  learning  as  a  dependent  measure  rather  than  reductions  in  training 
time. 

The  base  rate  data  should  be  supplemented  by  training  data  based  on  their  own  experiences 
using  training  technology.  Since  such  data  will  be  accumulated  gradually,  it  should  receive 
increasing  weight  as  more  data  is  collected  compared  to  general  population  data  collected  over 
larger  samples  and  larger  numbers  of  studies.  One  method  of  implementing  this  gradual  phasing 
in  of  organizational  data  with  base  rate  data  is  through  the  use  of  Bayesian  updating.  The 
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standard  Bayesian  updating  for  parameters  that  vary  from  zero  to  one  such  as  learning  rate  is 
based  upon  the  Beta  distribution.  The  Beta  distribution  is  characterized  by  two  parameters;  the 
success  rate  and  the  number  of  trials,  signifying  the  extensiveness  of  the  empirical  research. 

Each  update  then  adds  a  new  success  rate  and  the  number  of  trials  associated  with  this  data. 

These  data  are  combined  with  the  prior  information  to  produce  a  posterior  distribution.  Each 
new  update  is  added  to  all  of  the  previous  updates  and  the  prior.  As  more  and  more  data  are 
accumulated,  the  posterior  distribution  becomes  more  and  more  peaked  around  the  mean  with 
decreasing  variance. 

Based  on  the  literature  review  presented  in  section  2,  we  can  project  some  training  benefits  for 
training  the  IFF  task  using  different  media.  Note,  that  all  numbers  presented  are  in  mean  increase 
in  learning  compared  to  print-based  instruction  (which  has  a  baseline  benefit  of  0)  as  measured  in 
standard  deviations.  These  are  summarized  in  the  table  below. 


Tasks 

Print 

CBT 

IVD 

Internet 

Videotele- 

Conferencing 

Firing 

0 

.42 

.38 

N/A 

N/A 

IFF 

0 

.42 

.38 

N/A 

N/A 

Transponder 

Altitude 

0 

.42 

.38 

N/A 

N/A 

Speed 

0 

.42 

.38 

N/A 

N/A 

Safety 

— 

.42 

.38 

— 

N/A 

Corridor 


Table  10.  Projected  Training  Benefits  for  Training  Each  Cue  Using  Different  Media 

Notes:  The  symbol  ”  means  that  the  medium  is  not  acceptable  for  the  task.  “N/A”  means  we 
have  no  empirical  data  available.  Here,  a  trainer  could  insert  his/her  own  estimate  of  benefit.  In 
cases  where  multiple  data  are  available,  the  closest  population  to  the  target  Navy  population’s 
data  are  used  (e.g.,  military,  followed  by  adult,  etc.). 

3.5.2  Projection  of  costs. 

Having  developed  a  statement  of  training  benefit  under  each  training  delivery  option,  the  next 
step  is  to  compute  the  costs  of  each  option.  We  noted  earlier  that  there  are  two  broad  scenarios 
in  which  technology  may  be  adopted:  Conversion  of  existing  courses  to  technology  (in  which 
case  the  decision  is  likely  to  rest  on  whether  cost  savings  can  be  realized),  and  creation  of  new 
training  technology.  In  both  cases,  the  costs  of  technology  development  will  come  into  play.  In 
the  course  conversion  example,  the  costs  of  the  existing  course  will  also  need  to  be  calculated. 

Costs  of  existing  courses  will  need  to  be  calculated  by  the  trainer.  In  Phase  I,  we  developed  a 
Microsoft  Access-based  tool  to  help  CNET  perform  these  computations  for  existing  courses 
(discussed  in  section  4). 
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To  compute  technology  development  costs,  there  are  two  sources  of  data:  One  is  industry 
norms,  the  other  is  organization-specific  data.  Industry  norms  exist  on  how  much  different 
hardware  and  software  costs,  labor  rates  for  different  categories  of  professionals,  and  how  many 
hours  of  each  professional  are  required  to  develop  how  many  hours  of  instruction  on  different 
platforms.  Therefore,  industry  norms  could  be  used  exclusively  to  compute  how  much  it  would 
cost  to  develop  different  types  of  training  systems.  In  our  work  with  CNET,  we  learned  that  the 
Navy  works  through  different  contract  vehicles  where  equipment  purchases  and  labor  rates  are 
specified  contractually.  Therefore,  the  Navy  uses  these  assumptions  in  cost  estimating.  The 
unknown  variable  is  still  how  many  professional  hours  are  required  to  develop  the  technology. 

Again,  industry  norms  could  be  used.  On  the  other  hand,  the  Navy  may,  when  working  with  a  * 

consistent  set  of  contractors,  develop  a  stable  estimate  of  how  many  labor  hours  those  particular 
contractors  require  to  develop  technology.  The  Navy  could  use  those  estimates,  the  industry 
norms,  or  a  mix  of  the  two.  Because  there  is  uncertainty  in  costs  (as  well  as  training  benefit),  it  is 
useful  to  use  cost  ranges,  as  well  as  precise  numbers,  to  allow  for  imprecision  in  the  estimation 
process.  This  will  also  support  later  sensitivity/“what  if’  analyses. 

3.5.3  Conduct  cost-benefit  analysis. 

We  now  show  how  we  can  conduct  a  cost-benefit  analysis  using  the  benefits  computation  and 
the  costing.  For  benefits,  we  will  only  use  those  media  for  which  we  have  data.  For  costs,  we 
will  use  some  rough  estimates  supplied  to  us  in  our  work  with  CNET.  CNET  works  with  four 
levels  of  interactive  courseware.  The  second  level  of  ICW  (which  they  call  IB)  is  a  graphics 
presentation  with  text.  These  will  handle  all  the  IFF  cues  except  the  safety  corridor  which 
requires  a  high  fidelity  simulation.  The  fourth  level  of  ICW  (which  they  call  3)  is  a  high  fidelity 
simulation.  This  will  handle  the  safety  corridor. 

Cost  estimates  to  convert  an  hour  of  instruction  to  these  two  levels  of  ICW  are  $2500  and 
$8000,  respectively.  These  costs  do  not  include  requirements  and  task  analysis,  video  or  audio 
production  (which  there  is  likely  to  be  some),  development  tools,  and  media  distribution.  For 
purposes  of  our  running  example,  we  will  assume  that  the  costs  of  $2500  and  $8000  represent 
the  total  costs  of  building  the  training  technology  per  cue.  Therefore,  we  can  summarize  the 
costs  and  benefits  of  each  technology  option,  compared  to  a  baseline  of  existing  print-based 
instruction,  which  has  both  an  additional  cost  and  benefit  of  0. 
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Print 


Cues  Benefit  Cost 

Firing  0  0 

IFF  Transponder  0  0 

Altitude  0  0 

Speed  0  0 

Safety  Corridor 


Table  11.  Projected  Costs  and  Benefits  for  Training  Each  Cue  Using  Print 


ICW  Level  IB 

(low  level  CBT) 

Cues 

Benefit 

Cost 

Firing 

.42 

2500 

IFF  Transponder 

.42 

2500 

Altitude 

.42 

2500 

Speed 

.42 

2500 

Safety  Corridor 

— 

— 

Table  12.  Projected  Costs  and  Benefits  for  Training  Each  Cue  Using  ICW  Level  IB 


ICW  Level  3  (high  level  simulation-based  CBT) 


Cues 

Benefit 

Cost 

Firing 

.42 

8000 

IFF  Transponder 

.42 

8000 

Altitude 

.42 

8000 

Speed 

.42 

8000 

Safety  Corridor 

.42 

8000 

Table  13.  Projected  Costs  and  Benefits  for  Training  Each  Cue  Using  ICW  Level  3 
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Hybrid  Technology:  Level  IB  for  all  cues  except  corridor,  Level  3  for  corridor 


Cues 

Benefit 

Cost 

Firing 

.42 

2500 

IFF  Transponder 

.42 

2500 

Altitude 

.42 

2500 

Speed 

.42 

2500 

Safety  Corridor 

.42 

8000 

Table  14.  Projected  Costs  and  Benefits  for  Training  Each  Cue  Using  Hybrid  Technology 

The  above  tables  do  not  represent  the  continuing  costs  of  conducting  print-based  instruction. 
These  costs  only  represent  the  additional  development  costs.  If  course  conversion  was  being 
conducted,  then  the  ongoing  support  costs  for  delivering  print-based  instruction  as  well  as 
maintaining  and  updating  technology  would  have  to  be  included.  In  this  way,  trainers  could 
compute  whether  the  additional  cost  of  developing  technology  under  different  options  would 
ultimately  pay  for  itself  (CNET  identified  a  five  year  “break  even”  period  as  a  desirable  goal). 

We  illustrate  this  by  example:  The  cheapest  cost  of  course  conversion  that  captures  all  the 
learning  objectives  is  the  hybrid  technology  with  a  cost  of  $  1 8,000.  Let  us  assume  that  it  will 
reduce  training  time  by  30%  (consistent  with  findings  by  Orlansky  and  String,  1979  and  Fletcher, 
1991).  Let  us  assume  that  it  costs  $20,000  per  year  under  current  training.  For  purposes  of  this 
example,  we  will  assume  that  inflation  is  0  and  that  all  costs  are  tied  to  training  time  (there  are  no 
fixed  costs  that  occur  regardless  of  how  long  students  train).  Therefore,  it  will  cost  $14,000. 

The  following  table  illustrates  a  five  year  cumulative  dollar  cost  projection  for  both  course 
conversion  to  CBT  and  current  training  method: 


Year 

1 

2 

3 

4 

5 


Current  Approach  (cum  $) 
$20,000 
$40,000 
$60,000 
$80,000 
$100,000 


Conversion  to  CBT  (cum  $) 

$32,000 

$46,000 

$60,000 

$74,000 

$88,000 


Table  15.  Projected  Five  Year  Cumulative  Costs  for  Maintaining  Current  Course  Format 

and  Converting  to  Technology 

As  can  be  seen  from  the  above  table,  the  break  even  point  occurs  in  year  three  and  by  year 
five,  the  technology  has  saved  a  total  of  $12,000. 
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3.6  Conduct  tradeoff  analyses. 

It  is  clear  from  the  above  cost-benefit  tables  that  the  hybrid  technology  dominates  the  ICW 
level  3  option  as  it  delivers  the  same  training  benefit  at  reduced  costs.  However,  the  other 
options  are  less  clear.  In  all  cases,  money  can  be  saved  but  a  loss  occurs  in  training  benefit.  This 
is  where  the  benefit  of  the  INKS-based  operationalization  of  learning  objectives  comes  in.  INKS 
provides  a  mechanism  for  analyzing  not  only  what  topics  will  or  will  not  be  trained  by  each 
technology  option,  but  also  what  knowledge  and  skills  will  or  will  not  be  trained.  This  is 
illustrated  using  our  running  example. 

ICW  level  3  is  the  only  medium  that  trains  the  safety  corridor.  In  our  IFF  example,  recall  we 
had  the  following  INKS-based  inference  rules  (procedures)  associated  with  the  IFF  cues: 

a)  If  the  aircraft  is  firing  at  you,  treat  as  enemy. 

b)  If  the  aircraft  is  not  firing  at  you  and  its  IFF  transponder  signals  friendly,  treat 
as  friendly. 

c)  If  the  aircraft  is  not  firing  at  you  and  its  IFF  transponder  has  no  signal,  then  use 
a  two  out  of  three  rule  for  the  remaining  cues  to  determine  whether  to  treat  aircraft 
as  friend  or  enemy  (assuming  that  there  is  a  correspondence  between  altitude, 
speed  and  safety  corridor  and  what  friendly  vs.  enemy  planes  would  do). 

Using  this  knowledge,  we  can  see  that  if  we  opt  for  ICW  level  3,  all  rules  can  be  taught  using 
practice  on  the  simulation.  If  we  opt  for  ICW  level  IB,  we  cannot  teach  safety  corridor. 
Therefore,  we  can  still  teach  the  rule  about  enemy  firing,  we  can  teach  the  rule  about  IFF 
transponder,  and  we  can  teach  the  case  where  speed  and  altitude  both  signal  friendly  or  both 
signal  enemy  since  these  would  equal  the  two  out  of  the  three  necessary  to  make  the 
determination  of  IFF.  What  cannot  be  taught  is  the  case  where  altitude  and  speed  conflict  with 
each  other  since  safety  corridor  is  not  available  as  a  tie  breaker.  The  trainer  would  have  to 
determine  how  important  that  scenario  is  in  order  to  determine  what  tradeoff  to  make. 

Finally,  if  we  opt  for  print-based  media,  we  can  teach  all  the  cues  that  are  teachable  with  ICW 
level  IB.  This  sets  up  an  interesting  tradeoff  problem.  Assuming  we  eliminate  the  ICW  level  3 
option  because  it  is  dominated  by  the  hybrid  option  (same  training  benefit  as  the  ICW  level  3 
option  at  a  lower  cost),  we  have  three  remaining  options:  retain  the  print-based  instruction,  use 
ICW  level  IB,  and  use  the  hybrid  technology. 

Print  vs.  ICW  level  IB  is  a  case  where  both  media  meet  exactly  the  same  learning  objectives. 
The  main  difference  is  that  ICW  level  IB  produces  greater  learning  gains  but  for  a  higher  cost.  A 
decision  maker  would  have  to  make  the  choice  of  whether  it  was  worth  the  investment  to 
produce  these  higher  gains.  This  decision  may  be  driven  by  whether  the  organization  is  satisfied 
with  current  achievement  levels  of  trainees  who  receive  the  print-based  instruction.  ICW  level  3 
vs.  the  other  two  is  a  case  where  additional  learning  objectives  are  met,  but  at  a  higher  cost. 

Here,  the  tradeoff  is  whether  meeting  this  learning  objective  is  worth  the  estimated  $8000  it 
would  cost  to  build  the  technology.  Finally,  our  method  also  allows  for  innovative  solutions  in 
cases  of  limited  resources.  For  example,  the  decision  maker  could  opt  to  build  technology  for  the 
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safety  corridor  cue  but  retain  print-based  instruction  for  the  others.  This  option  would  cover  all 
learning  objectives,  although  the  learning  objectives  other  than  safety  corridor  would  be  covered 
via  the  medium  (print)  that  produces  the  least  amount  of  learning. 

We  cannot  “solve”  this  dilemma  in  the  abstract.  In  order  to  arrive  at  the  best  decision,  the 
organization  would  have  to  determine  how  satisfied  it  was  with  current  achievement  levels  so 
that  it  could  determine  the  added  value  of  additional  achievement  that  technology  could  offer.  It 
would  also  need  to  prioritize  its  learning  objectives  because  these  priorities  also  influence  the 
value  of  increased  learning.  Once  the  projected  value  of  the  increased  learning  was  established, 
the  organization  could  determine  whether  the  value  realized  was  worth  the  investment  required.  * 

Section  3.6.1  outlines  a  generic  tradeoff  methodology  that  describes  how  to  establish  these  values 
and  then  trade  them  off  against  costs. 

3.6.1  A  generic  tradeoff  methodology. 

The  design  of  a  multimedia  trainer  is  characterized  by  one  or  more  scarce  resources  (e.g., 

FY1999  “then-year”  dollars,  FY1995  person  hours)  that  have  to  be  “spread”  across  many 
components  of  the  multimedia  trainer  in  order  to  achieve  one  or  more  benefits  of  training.  Each 
component  needs  a  varying  amount  of  the  scarce  resource  in  order  to  provide  some  needed 
return,  but  typically,  insufficient  resources  are  available  to  fund  all  of  the  competing  components 
at  the  desired  level.  In  evaluating  the  worth  of  the  components,  measures  of  training  benefit  may 
be  focused  along  a  single  dimension  or  attribute,  or  they  may  be  so  complex  that  a 
“multiattribute”  model  of  training  benefits  is  needed.  In  order  to  solve  such  a  multimedia  design 
problem,  we  need  an  algorithm  that  insures  that  an  optimal  (or  nearly  optimal)  amount  of  training 
benefit  is  achieved  for  any  specified  resource  level  of  each  component. 

First,  we  will  consider  the  simplest  case  -  a  single  scarce  resource  needs  to  be  allocated 
across  a  finite  number  of  multimedia  components  having  a  single  measure  of  training  benefit. 

We  could  model  the  multimedia  components  as  continuous  functions  in  which  the  independent 
variable  is  the  resource  requirement  and  the  dependent  variable  is  the  benefit  returned.  This 
would  typically  be  solved  by  a  nonlinear  programming  algorithm  requiring  that  the  function  be 
concave  (e.g.,  first  derivative  continuous  and  positive  but  monotonically  nonincreasing). 

However,  we  have  found  that  very  few  people  think  of  their  system  components  in  this  way; 

rather,  they  identify  a  finite  number  of  discrete  funding  points,  called  “levels”,  for  each  „ 

multimedia  component.  Typically,  each  funding  level  has  very  compelling  rationale  for  what  is 

included  in  the  component  at  that  point  and  what  is  not.  In  the  simple  case,  we  will  model  each 

component  as  having  a  finite  number  of  funding  levels,  each  defined  in  terms  of  the  scarce 

resource  required  and  the  benefit  returned. 

This  model  of  a  multimedia  trainer  with  discrete  funding  levels  can  be  viewed  as  a  table  (see 
Figure  1)  in  which  each  row  represents  a  multimedia  component,  and  columns  represent  the 
funding  levels  within  a  row.  The  names  of  the  components  are  identified  on  the  left.  The 
column  labeled  “1”  defines  the  first  level  of  funding;  this  funding  level  is  the  minimum 
acceptable  level  of  funding  for  each  component.  In  a  zero-based  analysis,  the  funding 
requirements  in  each  component  would  be  zero.  However,  it  often  does  not  make  sense  to  do  a 
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zero-based  analysis  for  a  system  design,  in  which  case,  this  minimum  acceptable  must  be  defined 
in  an  agreed  upon  manner  by  the  trainers  and  technologists.  Each  succeeding  level  (or  column) 
in  a  specific  row  of  the  table  defines  a  funding  point  which  is  more  costly  than  the  previous  one. 
In  the  simplest  formulation,  each  succeeding  level  includes  all  previous  levels.  Note,  as  shown 
in  this  figure,  there  is  no  requirement  that  each  component  have  the  same  number  of  funding 
levels.  The  rightmost  level  in  each  row  represents  the  maximum  amount  of  funding  considered 
for  the  component  in  question.  We  allocate  our  resources  among  the  rows;  recall  that  there  is  not 
enough  resource  to  fund  the  rightmost  levels  for  all  components.  At  a  minimum,  there  must  be 
enough  of  the  scarce  resource  to  cover  the  first  levels  of  all  of  the  components  or  no  feasible 
"»  solution  will  exist. 
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Figure  1 .  Resource  Allocation  Matrix 

A  solution  to  this  multimedia  design  problem  can  then  be  defined  by  identifying  which  level 
in  each  row  obtains  funding.  The  shaded  boxes  in  Figure  1  show  one  possible  multimedia 
design. 

Since  the  scarce  resource  can  be  represented  in  a  single  dimension,  we  simply  have  to 
estimate  how  much  of  this  resource  is  required  to  satisfy  the  component  requirements  in  each 
level  of  each  row.  In  Figure  2  we  show  the  total  resource  needed  to  fund  the  component  at  each 
level  in  the  lower  left  comer  of  each  funding  level.  Since  these  numbers  are  defined  in  only  one 
dimension,  they  are  automatically  on  the  same  scale  and  can  be  compared  directly. 
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Figure  2.  Resource  Allocation  Matrix  with  Costs,  Scores,  and  Weights 

The  benefits  of  the  levels  must  also  be  estimated,  typically  by  educational  specialists  and 
technologists.  For  benefits,  we  often  do  not  have  a  readily  available  quantitative  measure  that 
can  be  used.  As  a  result,  we  have  to  create  one.  An  effective  approach  is  to  break  benefit  into 
two  components  —  a  score  measure  for  comparing  levels  within  a  single  component,  and  a 
calibration  measure  to  compare  benefits  across  components.  We  will  use  the  word  “scores”  to 
label  the  benefits  within  each  component,  and  the  word  “weights”  to  label  the  benefit  calibration 
measure  across  components.  For  ease  of  benefit  elicitation,  we  typically  normalize  the  scores  so 
that  the  benefit  associated  with  the  level  having  the  least  value  (usually  level  1)  is  zero  because  it 
is  a  baseline  representing  the  minimum  acceptable  funding  level.  In  addition,  the  benefit  score 
associated  with  the  most  valuable  funding  level  in  a  component  (typically  the  rightmost  level) 
will  be  100.  Note  that  we  could  have  chosen  any  arbitrary  endpoints  since  they  represent  relative 
benefit  measures.  Also  note  that  zero  does  not  mean  this  baseline  has  zero  benefit.  Rather,  it 
reflects  benefit  as  measured  from  a  baseline  level.  Similarly,  a  100  does  not  mean  that  all  needs 
are  met.  Rather,  it  reflects  100%  of  the  benefit  over  baseline  reflected  in  the  model.  The  benefit 
score  of  a  level  is  shown  in  the  lower  right  hand  comer  of  each  cell  in  the  matrix.  At  this  point  in 
time  in  the  modeling  process,  these  benefit  scores  cannot  be  compared  across  components.  To 
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make  such  a  comparison,  we  must  now  assign  weights  to  each  component  row  that  measure  the 
relative  benefit  of  “swinging”  from  minimum  to  maximum  funding  in  each  component;  this 
weight  is  shown  at  the  bottom  of  component  cell.  The  total  benefit  of  each  funding  level  is  then 
the  product  of  the  benefit  score  for  that  funding  level  multiplied  by  the  “swing”  weight  for  that 
component.  This  total  benefit  is  directly  comparable  across  components.  Figure  3  shows  a 
resource  allocation  model  with  the  costs,  scores  and  weights  placed  in  their  respective  positions. 

The  simplest  prioritization  rule  for  defining  which  component  levels  obtain  funding  is  the  one 
that  prioritizes  the  funding  levels  by  their  incremental  benefit  divided  by  their  incremental  cost 
>  (AB/AC),  typically  called  cost-benefit  analysis.  A  higher  benefit-to-cost  (lower  cost-to-benefit) 

ratio  is  better.  This  resource  allocation  algorithm  assumes  that  the  first  levels  of  each  component 
.  are  funded,  and  then  determines  an  “order-of-buy”  by  sorting  the  remaining  levels  from  highest 

to  lowest  by  their  incremental  benefit/cost  ratio  as  shown  in  Figure  3.  The  first  column  shows 
the  order  in  which  levels  are  funded,  with  a  0  indicating  that  a  level  is  in  the  baseline  (note:  the 
line  in  Figure  3  separates  the  baseline  from  the  trade  space).  The  “COMPONENT”  column 
names  the  row  which  has  the  next  level,  while  the  “LEVEL”  column  indicates  the  number  and 
name  of  the  next  level  funded.  The  next  four  columns  show  the  incremental  cost,  cumulative 
cost,  incremental  benefit,  and  cumulative  benefit,  respectively.  We  are  assuming  that  if  the  third 
level  of  the  first  component  is  funded,  then  levels  1  and  2  are  funded  as  well.  Having  defined 
this  order  of  buy,  we  can  next  tally  the  amount  of  resource  needed  to  get  to  each  succeeding  point 
on  the  priority  list.  There  will  come  a  point  where  we  can  afford  a  specific  level  on  the  list  but 
cannot  afford  the  next.  This  is  where  a  funding  line  will  be  drawn.  Everything  above  the  line  is 
funded,  and  everything  below  it  is  not. 


* 
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Figure  3.  Order-of-Buy 


The  results  of  this  order-of-buy  prioritization  can  be  plotted  in  two  dimensions,  cumulative 
benefit  versus  cumulative  resource  as  shown  by  the  dots  in  Figure  4. 
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Figure  4.  Benefit/Cost  Plot 

The  football  shaped  region  contains  all  of  the  possible  funding  options  and  is  known  as  the 
Pareto  space.  Starting  in  the  lower  left,  we  fund  only  the  minimum  acceptable  levels;  and  ending 
in  the  upper  right  comer,  we  have  enough  resource  to  fund  all  components  fully.  Our  scarce 
resource  constraint  will  fall  somewhere  between  these  two  extremes.  The  funding  solutions  that 
correspond  to  the  order-of-buy  fall  along  a  piece-wise  linear  upper  boundary  of  this  football, 
known  in  mathematical  programming  as  the  convex  hull.  This  makes  up  what  is  known  as  the 
Pareto  optimal  frontier  or  the  efficient  frontier. 

When  there  are  multiple  objectives  or  criteria  associated  with  the  competing  components,  the 
benefit  analysis  becomes  more  complex.  For  example,  with  conflicting  criteria,  a  component 
that  has  increasing  benefit  in  one  criterion  as  we  move  from  left  to  right  in  the  matrix  of  levels 
may  have  decreasing  benefit  in  another  criterion.  Examples  of  multiple  objectives  associated 
with  the  multimedia  trade-off  analysis  are:  training  time  for  fastest  20%  of  students,  training 
time  required  for  at  least  95%  of  class  to  pass  course,  and  operating  cost  per  year  (including 
technology  and  staff). 

There  are  several  equally  valid  ways  to  measure  and  compute  these  benefits.  For  the  purposes 
of  this  proposal,  we  will  just  present  one.  We  take  the  criteria  one  at  a  time  and  develop  a  0  to 
100  scale  for  each  row  on  each  criterion.  Since  the  criteria  may  conflict  with  each  other,  it  is 
likely  that  the  0  will  not  be  associated  with  the  first  level  and  the  1 00  will  not  be  associated  with 
the  rightmost  level  for  every  criterion.  This  will  change  some  of  our  assumptions  about  how  the 
levels  in  our  structure  table  are  defined;  however,  we  will  continue  to  assume  that  the  minimum 
acceptable  level  of  the  component  with  the  least  resource  requirement  is  in  the  first  level  and  that 
the  cumulative  expenditure  of  resources  increases  from  left  to  right. 
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Next,  we  will  define  “within  criterion”  weights;  we  will  select  a  single  criterion,  and  “within” 
the  selected  criterion,  we  will  assign  “swing”  weights  to  the  rows  as  before.  We  repeat  this 
procedure  for  each  criterion. 

Finally,  we  will  define  “across  criterion”  weights,  one  for  each  criterion  that  will  determine 
the  relative  impacts  of  the  criteria  on  the  results.  In  the  simplest  case,  we  can  think  of  these 
weights  as  the  relative  importance  of  the  criteria.  In  the  third  section,  we  will  interpret  these 
weights  in  a  more  mathematically  sophisticated  manner.  When  these  across  criterion  weights  are 
multiplied  by  the  within  criterion  weighted  scores,  we  have  the  final  benefit  measure  that  can  be 
compared  across  all  component  levels.  This  is  the  benefit  measure  that  will  be  used  in  the  * 

incremental  benefit/cost  analysis  to  determine  the  order-of-buy. 

Initially,  we  are  assuming  that  levels  are  ordered  in  decreasing  Abenefit/Acost  order; 
however,  there  is  no  guarantee  that  this  will  remain  the  case  if  we  change  the  across-criteria 
weights,  even  slightly.  Therefore,  whenever  any  of  the  across  criteria  weights  are  changed,  the 
structure  of  the  matrix  must  be  reexamined  before  the  resource  allocation  process  continues.  If 
the  within  criteria  weights  are  changed,  we  must  examine,  at  a  minimum,  those  components 
affected  by  the  change. 


4.0  DEVELOP  A  COSTING  TOOL  FOR  COURSE  CONVERSION. 

Purpose  of  Decision  Aid 

Currently,  curriculum  managers  of  courses  for  training  and  education  have  tens  to  hundreds  of 
courses  that  they  provide  on  a  periodic  basis,  primarily  via  the  traditional  one-on-many  lecture 
format.  There  is  continuing  pressure  to  deliver  this  material  more  cheaply  without  compromising 
the  educational  effectiveness  provided  to  the  student  population.  Unfortunately,  there  is  a 
substantial  cost  for  converting  lecture  material  with  overheads  to  even  the  most  basic  interactive 
courseware  (ICW)  with  a  single  thread  for  the  student  to  follow  at  her  own  pace.  Converting  to 
one  of  the  more  enhanced  interactive  courseware  options  with  multiple  branching  for  the 
students,  adaptive  testing  and  presentation,  sophisticated  simulations,  and  virtual  reality  requires 
even  greater  investments.  Competing  with  interactive  courseware  is  a  less  sophisticated  video 
teleconferencing  capability  that  allows  the  instructor  to  provide  the  lecture  material  to  multiple 
places  simultaneously.  This  format  primarily  saves  travel  time  for  the  students  or  the  instructor; 
the  up  front  cost  primarily  involves  hardware  and  software  for  delivering  the  course  to  multiple 
places.  The  potential  savings  for  interactive  courseware  are  far  greater;  the  instructor  can  be 
replaced  to  some  degree  by  the  courseware,  travel  costs  can  be  reduced  or  eliminated,  and  the 
students  can  use  the  courseware  anytime  and  anyplace,  providing  greater  flexibility  and  reduced 
time  for  many  of  the  students  as  well.  Finally,  slower  students  can  proceed  at  their  own  pace, 
making  it  possible  that  fewer  students  will  be  setback  or  dropped  for  academic  reasons. 
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Phase  I  Prototype  Activities 

As  part  of  the  phase  I  effort,  RDC  developed  a  prototype  decision  aid  for  examining  the 
relative  cost  of  conducting  a  given  course  using  the  standard  lecture  format  versus  one  of  four 
Interactive  Courseware  (ICW)  delivery  options  and  a  Video  Teletraining  (VTT)  option.  The  user 
of  such  a  decision  aid  would  be  curriculum  managers.  These  managers  are  either  trying  to  save 
money  while  maintaining  a  given  level  of  training  effectiveness  or  enhancing  training 
effectiveness  while  maintaining  a  fixed  level  of  spending.  Our  prototype  only  deals  with  the  first 
objective,  saving  money  while  maintaining  effectiveness.  The  purpose  for  this  prototype  was  to 
i  obtain  feedback  from  curriculum  managers  on  what  features  they  needed  in  such  a  decision  aid. 

The  four  types  of  ICW  were  based  on  the  military  handbook  for  ICW :  ( 1  A)  linear  video 
presentation  with  minor  text;  (IB)  linear  graphics  presentation  with  minor  text;  (2)  medium 
simulation  with  computer  managed  instruction;  and  (3)  high  level  simulation  permitting  a  high 
level  of  student  interactivity,  extensive  branching  capability,  maximum  remediation 
opportunities,  and  real  time  event  simulation.  A  summary  of  the  features  for  these  ICW  options 
is  shown  in  the  table  below.  The  estimated  costs  to  convert  an  hour  of  lecture  and  other  material 
to  ICW  format  are  $2000,  $2500,  $4500,  and  $8000,  respectively.  These  cost  estimates  do  not 
include  requirements  and  task  analyses,  video  or  audio  production,  development  tools,  or  media 
distribution. 


Capabilities 

ICW  1A  and  IB 

ICW  2 

ICW  3 

Interactivity 

Low 

Medium 

Extensive 

Sequence  Control 

Low 

Medium 

Extensive 

Simulation 

None 

Simple  Processes 

Complex,  Real  Time 

Branching 

Low 

Medium 

Extensive 

Remediation 

None 

Medium 

Extensive 

Feedback 

Low 

Medium 

Extensive 

Management  System 

None 

Medium 

Extensive 

Gaming 

None 

None 

As  requested 

Video 

Basic 

Multimedia 

Multimedia 

Text 

Yes 

Yes 

Yes 

Graphics 

Yes 

Yes 

Yes 

Audio 

Yes 

Yes 

Yes 

Table  16.  Capabilities  of  Different  Levels  of  ICW 

The  prototype  aid  computes  development  costs,  delivery  costs  and  travel  costs  for  both 
current  course  delivery  and  the  ICW/VTT  options.  The  development  of  ICW  or  VTT  will 
generally  reduce  travel  costs  and  some  delivery  costs;  other  delivery  costs  (e.g.,  communication 
costs)  may  go  up.  These  costs  were  estimated  for  a  five  year  time  horizon. 

The  prototype  of  this  decision  aid  developed  by  RDC  during  the  phase  I  SBIR  effort  used 
Microsoft’s  ACCESS  database  management  software  with  a  Visual  Basic  user  interface.  This 
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prototype  provides  the  basic  functionality  needed  by  a  curriculum  manager,  but  is  a  very  simple 
version  of  the  ultimate  version  of  the  decision  aid  that  is  needed  by  a  curriculum  manager. 

The  user  can  enter  information  about  a  given  course  concerning  the  numbers  of  students  and 
instructors  per  offering,  the  length  of  the  course,  the  number  of  offerings  per  year,  the  travel 
associated  with  the  course,  the  fixed,  and  variable  costs  associated  with  the  course.  The  software 
prototype  then  provides  a  yearly  cost  summary  of  (1)  continuing  the  current  approach  to  the 
course  offering,  (2)  converting  to  various  ICW  options  for  providing  the  course  and  (3)  using 
video  teleconferencing  to  provide  the  course.  This  cost  summary  shows  whether  the 
development  costs  for  the  ICW  or  video  teleconferencing  provide  sufficient  cost  savings  over  a  r 

five  year  period  to  provide  savings  over  and  above  the  development  costs. 


5.0  TEST  THE  TOOL  ON  CNET  SUPPLIED  COURSES. 

We  tested  this  prototype  on  four  CNET  courses: 

1 .  Basic  Submarine  Torpedo  Supervisor 

2.  Advanced  Electronics  and  Maintenance  Techniques 

3 .  Gun  F ire  Control  System,  Mod  8/ 1 0 

4.  Type  Al,  FC  specific  electronics  training 

This  prototype  was  demonstrated  to  managers  of  education  and  training  courses  for  the  Navy. 
After  the  demonstration,  these  managers  fiimished  us  with  data  necessary  for  computing  the  costs 
of  converting  the  courses.  Such  data  are  described  in  Section  4.0  above,  and  include  variables 
such  as  the  number  of  trainees  per  course,  travel  costs,  instructor  costs,  etc. 

Once  we  completed  our  analyses,  we  returned  to  CNET.  The  course  of  highest  priority  to 
them  was  the  Basic  Submarine  Torpedo  Supervisor  course.  Therefore,  the  demonstration  of  the 
tool’s  analysis  of  CNET  supplied  data  focused  on  this  course.  After  some  initial  data  exchange 
and  verbal  discussion,  the  prototype  generated  recommendations,  based  on  cost  savings,  for 
which  technology  medium  to  use  in  the  course  conversion.  These  recommendations  were  the 
same  as  the  conclusions  that  CNET  had  reached  when  they  performed  their  own  independent 
analysis.  (The  results  of  this  analysis  are  actually  shown  in  Figure  17.) 


6.0  DISCUSSION 

The  present  project  attempted  to  fill  a  need  within  the  training  and  education  communities 
regarding  how  technology  is  purchased  and  used.  In  practice,  much  technology  procurement  is 
done  ad  hoc  without  a  formal  analysis  of  what  the  projected  training  benefits  will  be  measured 
against  the  costs  involved.  The  main  innovation  offered  in  the  present  project  is  the 
methodology  for  operationalizing  a  set  of  learning  objectives  in  such  a  way  as  to  specify 
technology  requirements,  and  in  turn,  training  benefits  that  are  expected  from  different 
technology  options.  Such  a  methodology  allows  the  training  community  both  to  insure  that  the 
technology  it  procures  will  be  adequate  for  the  requirements  yet  not  cost  more  than  is  necessary. 
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Because  the  issue  of  moving  training  from  the  schoolhouse  to  technology  is  such  a  hot  issue, 
there  are  several  other  efforts  within  the  Navy  to  develop  decision  making/analysis  approaches 
for  using  multimedia  technology  for  training.  We  present  illustrative  projects  and  highlight  the 
differences  and,  hopefully,  value-added  to  our  approach.  In  these  discussions,  we  present  the 
caveat  that  our  understanding  of  all  the  work  being  done  may  be  incomplete  as  efforts  continue  to 
be  ongoing. 

6.1  Differences  between  our  approach  and  one  being  developed  by  CNET 

In  our  work  with  CNET,  we  learned  that  they  are  interested  in  converting  courses  from 
classroom-based  to  technology-based.  Their  stated  objective  is  to  justify  the  costs  of  doing  so  by 
demonstrating  a  projected  break-even  point  within  five  years  of  making  the  investment  to 
develop  the  technology.  CNET  was  less  focused  on  developing  models  to  predict  training 
benefits  of  different  training  technologies.  They  correctly  observed  that  the  literature  reporting 
training  benefits  tended  to  show  variable  results.  Therefore,  they  feel  less  comfortable  using 
such  inputs  in  their  analyses. 

On  the  other  hand,  we  know  of  no  formalism  within  their  work  that  specifies  which 
technology  to  convert  a  course  into  (e.g.,  CAI,  ICW,  VT).  This  appears  to  be  done  as  a  matter  of 
professional  judgment  or  perhaps  driven  by  cost  factors. 

We  believe  our  framework  provides  an  analysis  method  for  selecting  which  media  to  use  for 
training  a  course.  The  framework  provides  an  empirical  justification  for  projecting  training 
benefits  of  those  media,  even  if  such  benefits  may  have  to  be  expressed  as  a  projected  range  of 
benefits  as  opposed  to  a  precise  benefit.  However,  cognizant  of  the  imprecision  in  the  estimation 
process,  we  provide  two  remedies.  First,  we  discuss  sensitivity  analysis  to  explore  different 
assumptions  regarding  those  benefits  in  a  cost-benefit  analysis.  Second,  we  discuss  the  use  of 
data  collection  and  Bayesian  updating  tools  to  allow  the  users  to  collect  training  benefit  data  on 
their  own  population  of  students,  courses  and  technologies,  which  we  believe  will  lead  to  more 
reliable  and  stable  projections  of  training  benefits. 

We  cited  earlier,  however,  that  there  was  a  second  scenario  in  which  training  technology  may 
be  developed,  namely  development  of  training  for  new  technologies/platforms/requirements,  etc. 
In  such  cases,  an  existing  successful  course  is  likely  to  be  unavailable.  As  far  as  we  can  tell,  the 
CNET  framework  is  silent  on  which  training  technology  to  use.  In  such  cases,  there  is  likely  to 
be  additional  media  not  considered  by  their  current  work,  e.g.,  embedded  training,  Internet-based 
training,  Distributed  Interactive  Simulation,  immersive  virtual  reality. 

We  believe  that  if  the  Navy  is  to  exploit  the  proliferation  of  technologies  available  to  deliver 
training,  there  needs  to  be  analysis  methods  with  a  theoretical/empirical  basis  to  help  select 
which  of  these  media  will  deliver  the  intended  training.  We  believe  this  needs  to  be  taken  into 
consideration  as  well  as  costing. 
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We  agree  with  comments  made  by  Dr.  Jim  Young  of  NAVAIR  that  building  such  a 
theoretical/empirical  framework  will  take  years  in  the  making  and  would  not  be  responsive  to  the 
immediate  needs  of  converting  courses  to  technology-based  training.  However,  the  issue  of  what 
training  to  accompany  what  need  (given  that  there  are  more  and  more  options  to  deliver  the 
training)  is  a  long  term  problem.  The  present  work  is  designed  to  address  the  immediate  course 
conversion  needs  as  well  as  provide  a  foundation  for  addressing  these  long  term  problems. 

6.2  Differences  between  our  approach  and  TRADAM. 

The  Defense  Standards  Workgroup  has  recently  developed  a  Training  Delivery  Assessment  r 

Model  (TRADAM).  TRADAM  is  a  fairly  comprehensive  analysis  process  that  takes  a  training 
requirement  and  analyzes  it  through  various  steps  of  developing  learning  objectives,  defining 
sensory  cues,  media  selection,  training  system  design,  etc.  As  such,  we  see  TRADAM  as  a 
useful  methodology  for  addressing  the  need  described  in  the  previous  section,  namely  matching 
future  training  requirements  to  the  media  most  suitable  for  selection.  As  described  currently,  it  is 
less  detailed,  although  consistent  with,  the  costing  models  that  organizations  such  as  CNET  need 
for  course  conversion  analyses. 

We  see  our  framework  as  highly  compatible  with  TRADAM.  Our  framework  stops  at  the 
point  of  selecting  the  training  media,  analyzing  the  costs  and  benefits  of  different  options  and 
then  conducting  a  tradeoff  analysis  and  making  decisions  regarding  resource  allocation.  While 
our  analysis  would  then  support  the  design  and  maintenance  of  the  technology,  our  framework 
does  not  prescribe  how  to  do  so.  (We  could  extend  our  framework  accordingly,  as  we  are 
experienced  training  system  designers.  However,  we  see  this  as  being  beyond  the  scope  of  the 
SBIR  topic  to  which  we  are  responding.) 

We  believe  our  work  enhances  TRADAM  in  several  respects.  Based  on  discussions  with  Dr. 

Bill  Terrell  of  NAWC  TSD,  we  have  identified  two  value  added  areas  the  proposed  project  will 
address.  First,  Dr.  Terrell  noted  that  TRADAM  does  not  provide  an  analysis  of  how  to  help  the 
student  learn.  In  other  words,  media  selection  is  derived  from  the  sensory  cures  necessary  to 
meet  a  learning  objective.  There  is  not  a  link  between  the  media  and  the  cognitive  requirements 
for  learning  the  objective.  There  is  no  empirical  database  for  justifying  the  media  selection. 

There  is  no  methodology  for  specifying  the  mix  of  instruction  to  build  specific  concepts.  In 

Section  3.3,  we  describe  a  step  in  our  methodology,  “Develop  training  strategy”,  which  is  ^ 

missing  in  the  TRADAM.  This  step  occurs  prior  to  media  selection  and  drives  that  process. 

A  second  need  identified  by  Dr.  Terrell  was  tools  to  support  the  implementation  of 
TRADAM.  Dr.  Terrell  expressed  the  concern  over  whether  members  of  the  training  community 
would  adopt  TRADAM  if  no  tools  were  available  to  help  the  process.  A  future  direction  our 
work  would  take  would  be  to  develop  automated  tools  to  implement  our  framework,  which  we 
stated,  was  an  enhanced  subset  of  the  TRADAM. 

We  stated  above  that  although  TRADAM  was  consistent  with  the  cost  analysis  requirements 
of  technology  procurement,  it  was  less  detailed  than  is  needed  by  an  organization  such  as  CNET 
to  perform  such  analysis.  Recognizing  that  course  conversion  is  an  immediate  need  faced  by  the 
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Navy  (as  well  as  other  organizations),  our  work  is  designed  to  support  the  in-depth  cost  analysis 
required  for  this  process. 

By  comparing  the  above  two  ongoing  efforts  (CNET,  TRAD  AM),  we  have  tried  to  show  that 
each  has  strengths  and  areas  for  value-added  enhancement.  Specifically,  each  approach  appears 
to  be  better  suited  for  one  of  the  scenarios  of  training  technology  development  (i.e.,  CNET  for 
course  conversion,  TRADAM  for  long  term  training  system  development).  The  long  term  goal 
of  our  work  is  to  develop  a  methodology  and  supporting  tool  to  support  both  needs  and  enhance 
the  ongoing  work  in  each  community. 


7.0  FUTURE  DIRECTIONS 

The  biggest  benefit  to  the  training  community  should  come  in  terms  of  developing  automated 
tools  to  simplify  the  types  of  analyses  described  in  Section  3.  The  tool  needs  to  do  more  than 
simply  step  the  user  through  the  complexities  of  the  decision  making  process  (e.g.,  handling  on 
the  technology-relevant  data,  performing  cost-benefit  and  tradeoff  computations).  It  is  important 
that  the  tool  help  the  user  maintain  continuity  in  the  thought  process.  As  in  all  training  and 
education  technology  procurements,  the  goal  is  to  use  technology  to  support  learning.  However, 
it  appears  that  technology  is  often  bought  without  a  formal  analysis  of  how  that  technology  will 
enhance  student  learning. 

Our  project  has  tried  to  remedy  this  process  by  creating  a  methodology  that  starts  with 
organization-defined  learning  objectives.  These  learning  objectives  form  the  basis  for 
developing  technology  requirements  that  will  meet  these  objective.  The  cost-benefit  analysis 
also  ties  directly  back  to  the  learning  objectives.  Tradeoff  analysis  presents  a  picture  not  only  of 
monies  that  can  be  saved  but  also  learning  objectives  that  are  impacted  by  that. 

Below  we  describe  the  design  of  a  more  comprehensive  tool  that  will  implement  our 
methodology  while  preserving  the  continuity  of  the  learning  objective-driven,  decision-making 
process. 

7.1  An  automated  tool  to  support  our  TRAD  AM-enhanced  methodology. 

Our  goal  is  to  support  the  both  course  conversion  analysis  and  the  TRADAM  process.  With 
regard  to  the  former,  we  have  a  leg  up  as  the  Phase  I  system  was  designed  to  do  that  and  was  far 
enough  along  as  to  provide  analysis  for  sample  CNET  courses.  With  regard  to  the  latter,  the  tool 
will  not  be  a  one-to-one  match  to  TRADAM  (but  will  be  a  one-to-one  match  to  our 
methodology)  as  it  will  omit  some  steps  in  the  overall  process  while  adding  the  step  of 
developing  a  learning  strategy. 

System  Architecture 

The  design  of  this  decision  aid  contains  six  basic  elements  as  shown  in  Figure  5.  The  top 
element  controls  all  interaction  with  the  user  via  keyboard  and  mouse  inputs  and  well  as  the 
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displays  on  the  screen.  The  left  element  maintains  the  various  databases  of  the  users  as  well  as 

the  databases  for  costs,  technology,  training  effectiveness  supplied  with  the  decision  aid.  The 

two  middle  elements  perform  the  processing  required  by  the  analyses  and  control  this  processing, 

as  well  as,  the  user  interaction,  database  operations,  and  report  generation.  The  Cost  Analysis 

module  is  part  of  the  processing  element  and  is  responsible  for  performing  cost  analysis  on  the 

course  data  for  each  of  the  possible  course  formats  (Current  IM,  ICW-1  A,  VT,  etc.).  The  Benefit 

Analysis  module  is  also  part  of  the  processing  element,  differs  only  slightly  in  form  and  function 

from  the  Cost  Analysis  module,  and  is  responsible  for  performing  a  benefit  analysis  of  the  course 

in  its  various  formats.  The  bottom  element  contains  the  help  database  as  well  as  provides  a 

repository  for  historical  data  based  upon  usage  of  the  decision  aid  over  a  long  time  period.  * 

Finally,  the  right  element  is  responsible  for  providing  hard  copy  reports  to  the  users. 


User  Interface  Processing 

Data  Base 
Importing 
& 

Maintenance 

Process  Model 

Report 

Generation 

Control  Model 

Historical  Data, 
Help  Support 

Figure  5.  Design  Elements  of  Decision  Aid 


The  following  figures  and  descriptive  material  describe  samples  of  the  system  architecture  as 
we  currently  envision  them. 
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Customization  of  such  elements  as  the  time  period  and  cost  elements  by  the  user: 


Figure  6.  Customization  by  the  User 

The  user  shall  be  able  to  customize  the  decision  aid  based  upon  the  relevant  time  periods  for 
her  organization.  For  example,  the  user  shall  be  able  to  indicate  that  her  organization  keeps  cost 
elements  by  hour,  day,  week,  month,  or  year.  In  addition,  the  user  will  be  able  to  identify  certain 
groupings  for  fixed  and  variable  cost  elements.  These  entries  will  determine  how  the  user’s 
database  is  set  up  and  will  cause  conversions  to  the  default  cost  data  base. 
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Personalization  of  the  evaluation  process: 


Figure  7.  Evaluation  Personalization 


The  decision  aid  shall  permit  the  user  to  personalize  specific  aspects  of  how  the  analyses  will 
be  conducted.  For  example,  the  user  shall  be  able  to  select  whether  a  course  will  be  evaluated  on 
its  own  merit  or  whether  there  is  a  budget  for  converting  courses  and  each  selected  course  is 
competing  for  a  portion  of  that  budget.  In  addition,  the  user  will  be  able  to  enter  her  objective; 
namely  maximize  cost  savings  or  maximize  training  or  educational  effectiveness.  Finally,  the 
user  shall  be  able  to  indicate  whether  all  courses  will  only  be  considered  for  complete  conversion 
or  whether  it  is  possible  to  convert  portions  of  the  courses.  If  the  latter  is  chosen,  the  user  must 
indicate  what  criteria  will  be  used  for  converting  portions  of  a  course  (e.g.,  labs  may  be  converted 
before  the  lectures). 
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Personalization  of  the  results  and  display  visualization: 


Figure  8.  Personalized  Visualization 


The  decision  aid  shall  permit  the  user  to  personalize  the  visualization  of  results.  This 
personalization  will  address  the  types  of  charts  to  use  to  display  each  category  of  results.  In 
addition,  the  user  shall  be  able  to  choose  whether  to  select  the  display  of  certain  combinations  of 
results  on  the  screen  simultaneously. 


A 
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Conducting  a  analysis  of  conversion  options  based  on  one  of  several  objectives: 
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Figure  9.  Conversion  Analysis 


The  decision  aid  shall  permit  the  user  to  enter  the  specific  course(s)  to  be  analyzed  and  the 
objectives  and  assumptions  that  should  be  made  as  part  of  the  analysis.  Based  on  these  entries 
the  decision  aid  shall  then  complete  the  analysis  by  conducting  either  a  cost,  benefit  or  cost- 
effectiveness  analysis  of  each  option  as  well  as  the  current  course  delivery  option.  In  addition, 
the  decision  aid  shall  analyze  the  relative  training  effectiveness  of  each  option  (if  so  instructed  by 
the  user).  The  training  effectiveness  analysis  shall  include  training  effectiveness  data  based  upon 
empirical  analyses  conducted  by  the  research  community.  This  training  effectiveness  analysis 
will  include  Cognitive  Structure  Analysis  using  the  INKS  framework  described  elsewhere  in  the 
proposal.  These  results  will  be  stored  in  a  user  database  as  identified  by  the  user. 


48 


Technical  Report  97-012 


Display  of  analysis  results  to  the  user: 


Figure  10.  Display  of  Results 


The  user  shall  be  able  to  obtain  a  display  of  the  results  of  an  analysis  of  one  or  more  course 
conversions  by  selecting  the  results  display  menu  option.  The  decision  aid  will  create  displays 
based  upon  the  currently  selected  display  criteria;  the  user  has  the  option  to  change  these  criteria 
as  describe  above.  In  addition,  to  displaying  the  relative  attractiveness  of  the  competing  course 
conversion  options  on  whatever  objectives  defined  by  the  user,  the  decision  aid  will  also  display 
sensitivity  analyses  of  these  results  to  specifically  identified  parameters  defined  by  the  user. 
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Provision  of  help  to  the  user: 


Figure  11.  Help  Menu 


The  user  shall  be  able  to  obtain  full  on-line  help.  There  will  be  a  instructions  on  how  to  use 
each  menu  selection,  hypertext  glossary  of  terms  and  results,  drill  down  explanation  of  results, 
and  explanation  of  sensitivity  analyses. 
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Database  updating  and  creation/maintenance  of  a  historical  database: 


Figure  12.  Database  Updating  and  Backup 

The  decision  aid  shall  permit  the  user  to  update  original  databases  with  more  recent 
information,  as  well  as  to  back  up  all  of  the  databases  to  storage  media.  The  user  will  select  the 
database  menu  item  and  identify  whether  she  wishes  to  update  the  databases  or  back  up  her 
databases.  In  addition,  it  will  be  possible  to  update  the  current  historical  database  with  more 
recent  users’  files. 
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Generation  of  reports  for  the  user: 


Figure  13.  Report  Generation 


Reports  that  document  the  results  of  a  user’s  analysis  shall  be  generated  when  the  user  selects 
the  report  generation  menu.  The  user  will  identify  which  standard  report  section  should  be 
included  and  what  types  of  graphics  (e.g.,  pie  charts,  bar  charts)  to  include.  Based  on  these 
inputs  the  decision  aid  will  generate  the  report  consistent  with  the  data  in  the  user’s  database. 
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Provision  of  historical  analyses  that  were  supported  by  the  decision  aid: 


Figure  14.  Historical  Analyses 

The  decision  aid  shall  permit  historical  analyses  to  be  performed  based  upon  a  large  number 
of  uses  by  one  or  more  users  over  a  long  period  of  time.  The  user  selects  the  historical  analysis 
menu  item,  identifies  what  elements  of  the  historical  analysis  should  be  included,  identifies 
which  user  databases  are  to  be  included,  and  then  turns  the  creation  of  the  historical  analysis  over 
to  the  decision  aid.  The  historical  analysis  addresses  what  trends  have  been  observed  over  the 
horizon  of  the  analysis,  including  trends  in  costs  and  trends  in  changes  to  course  conversion 
selections. 
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System  Interface 

When  a  course  is  initially  entered  into  the  system,  there  are  several  key  pieces  of  information 
that  must  be  entered.  The  course  must  be  identified  by  a  unique  name  such  that  it  can  be 
retrieved  at  a  later  date.  The  information  necessary  for  performing  the  cost  analysis  is  entered 

through  a  series  of  screens. 

The  screens  prompt  the  user  to 
enter  data  through  a  series  of 
questions  about  the  course. 
Most  of  these  questions  are 
very  direct  and  difficult  to 
misinterpret.  Some  of  the  data 
will  be  known  ahead  of  time, 
such  as  the  cost  for  someone 
to  teach  the  course.  Other  data 
will  not  be  concrete  and 
instead  need  to  be  estimated, 
such  as  the  cost  for  developing 
the  software  necessary  to 
convert  a  lecture  to  a  CBT.  However,  industry  standards  should  exist  for  most  of  these  values 
and  can  be  stored  and  modified  in  the  Database  Management  module.  Thus,  the  system  can  use 
values  which  can  be  stored  and  modified  in  the  Database  Management  module.  The  system  can 
use  default  values  or  the  user  can  override  these  numbers  if  they  have  more  accurate  estimates. 

The  data  necessary  to  perform  the  course  benefit  analysis  is  entered  in  a  similar  fashion.  The 
system  prompts  the  user  (usually  an  SME  or  similarly  qualified  person)  to  break  the  course  down 
into  a  series  of  learning  objectives.  Each  objective  is  broken  further  into  a  series  of  components 
and  their  INKS  representation.  Through  a  series  of  check  boxes,  the  system  then  prompts  the 
user  to  detail  the  experiential  breakdown  of  these  components,  be  it  audio,  video,  low  fidelity, 
high  fidelity,  mathematical,  temporal,  spatial,  etc.  By  breaking  the  objective  components  down 
into  their  experiential  representation,  the  system  can  determine  the  type  of  media  the  course 
requires  to  teach  the  students.  For  example,  if  a  course  objective  component  had  an  experiential 
representation  consisting  of  audio,  visual,  temporal,  spatial,  and  high  fidelity,  it  may  indicate  to 
the  system  that  a  VR  chamber  is  necessary  to  teach  the  course  objective.  However,  a  visual,  low 
fidelity  requirement  may  indicate  a  simple  picture.  The  system  then  compares  this  information  to 
the  target  student  population’s  profile  for  learning  methods,  and  a  course  benefit  analysis  is 
made.  This  course  benefit  analysis  will  supply  the  user  with  the  characteristics  of  the  format  or 
formats  necessary  to  teach  the  course.  The  profile  of  the  target  student  population  is  not  input 
directly  by  the  user.  Instead,  he  chooses  from  a  menu  of  profiles  already  in  the  system.  There  is 
also  the  option  for  the  user  to  modify  these  existing  profiles,  as  well  as  input  new  ones  as 
necessary.  However,  these  are  done  separate  from  the  course  analysis. 
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The  help  offered  will  be  context- 
sensitive  so  that  the  user  can  query  a 
particular  button  or  question  that  is  on  the 
screen  and  see  the  actual  results.  For 
example,  querying  a  bold  button  in  a  word 
processor  gives  the  user  a  brief  explanation 
of  what  actions  the  button  performs. 

Similarly,  querying  a  question  in  the  tool 
1  will  result  in  a  more  detailed  explanation  of  Figure  16.  Context-sensitive  help 

what  information  is  being  sought.  Context- 

*  sensitive  help  will  increase  user  productivity  by  increasing  understanding  of  the  tool  and  how  to 

use  it. 

The  results  of  the  analyses  can 
be  viewed  in  several  formats.  The 
graphical  representation  consists  of 
a  2-dimensional  cost  vs.  time  bar 
graph  for  the  cost  analysis,  where 
each  line  will  represent  a  particular 
course  format.  A  bar  graph 
displaying  the  relative  benefit  for 
each  of  the  possible  course  formats 
shows  the  results  of  the  benefit 
analysis.  A  scatter  plot  showing 
the  training  benefit  vs.  cost  for  a  Figure  17.  Graph  of  Cost  Analysis 

specified  period  of  time  displays 

the  results  of  the  cost-benefit  analysis.  By  examining  this  chart,  the  user  will  be  able  to 
determine  the  optimal  course  format  that  fits  under  their  financial  constraints. 

As  was  previously  mentioned,  one  of  the  benefits  of  our  architecture  is  the  interactive  analysis 
that  it  allows.  When  the  system  performs  a  course  benefit  analysis,  it  presents  the  user  with  a  list 
,  of  features  of  the  course  system  requirements.  This  presentation  is  modifiable  so  that  the  user 

can  set  certain  constraints.  For  example,  if  there  is  no  way  the  budget  will  allow  for  a  VR 
chamber,  then  the  user  can  downgrade  the  allowable  system  features.  The  program  will  then  go 
back  and  determine  which  of  the  objective  components  are  no  longer  satisfied  and  present  a  list 
of  them  to  the  user.  It  will  also  present  the  user  with  the  course  objective  or  objectives  affected 
and  a  brief  explanation  detailing  why  each  objective  is  no  longer  possible  as  originally  described 
by  the  user.  Armed  with  this  tool,  the  user  will  be  able  to  customize  the  course  to  maximize  the 
benefit  to  the  student  population  while  staying  within  the  financial  and  physical  constraints  of  the 
course. 
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7.2  Potential  Research  Questions  Suggested  by  Our  Framework 

There  are  still  be  several  areas  of  further  R&D  that  could  be  performed.  First,  the  biggest 
weakness  in  the  proposed  framework  is  to  project  training  benefits  given,  the  topic  matter  being 
taught,  and  the  media  selected.  As  we  have  noted  earlier,  literature  consistently  shows  that 
technology  can  enhance  learning  achievement  or  reduce  training  time.  We  also  hypothesized  that 
training  organizations  would  achieve  more  stable  benefits  predictions  if  they  collected  data  on 
their  own  trainee  populations,  courses  taught,  and  technologies  used.  While  there  will  always  be 
such  variability,  we  believe  this  can  be  reduced.  Clearly,  an  area  of  future  R&D  is  to  collect  data 
against  this  hypothesis.  The  benefit  for  an  organization  in  doing  so  is  to  build  an  experiential  f 

base  of  training  technology  usefulness. 

♦ 

Part  of  this  R&D  effort  could  focus  on  the  specific  analysis  provided  by  the  INKS  framework 
compared  to  other  analyses  suggested  by  other  frameworks.  For  example,  our  methodology  uses 
INKS  and  a  combination  of  sensory  cues  to  develop  a  learning  strategy  to  select  training  media. 

TRAD  AM  relies  more  on  a  sensory  cues  analysis  derived  from  learning  objectives  analysis. 

CNET  uses  professional  judgment.  Each  of  these  three  methods  (plus  others)  could  be  pitted 
against  each  other.  A  study  might  involve  taking  different  subject  areas  (to  test  robustness  of  the 
methods)  and  performing  a  media  selection  analysis  for  each.  Groups  of  trainees  would  be 
selected  based  on  the  media  recommended.  Training  time  and  performance  could  be  measured  to 
determine  which  method  offered  the  most  effective  media  selection. 

Another  area  touched  upon  in  our  framework  was  inclusion  of  student  population 
characteristics  in  the  training  strategy  selection  process.  There  is  considerable  work  in  learning 
styles/leaming  disabilities  that  offer  input  as  to  what  training  media  may  be  most  effective.  We 
also  touched  upon  experience  and  aptitude  as  relevant  factors.  There  is  considerable  room  for 
further  research  in  exploring  these  individual  difference  variables  in  the  use  of  training 
technology. 

Related  to  the  issue  of  including  student  parameters  in  the  selection  process  is  assessment  of 
student  learning.  One  of  our  parameters  is  student  experience.  This  could  include,  not  only 
prerequisite  courses  taken,  but  also  proficiency  achieved  in  those  courses  (which  might  also  be 
dependent  upon  aptitude).  Linking  assessment  tools  to  the  media  selection  process  could  provide 
useful  input  regarding  a  student’s  baseline  knowledge  which  in  turn  would  develop  the  training 
strategy.  We  are  currently  working  on  a  Phase  I SBIR  project  sponsored  by  NAWC  TSD  to 
develop  assessment  tools  for  complex  cognitive  tools.  Assessment  is  an  area  of  specialization 
for  RDC  (we  were  retained  by  the  Point  Pleasant  School  District  in  New  Jersey  to  help  them 
develop  their  assessment  framework  based  on  our  INKS  model).  We  envision  such  assessment 
tools  as  ultimately  integratable  into  our  proposed  tool. 


8.0  POTENTIAL  COMMERCIAL  APPLICATIONS  OF  THE  PRESENT  WORK 

The  present  project  was  sponsored  under  the  Small  Business  Innovation  Research  program.  A 
goal  of  this  program  is  to  sponsor  R&D  activities  that  result  in  commercial  products  and  services. 
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We  believe  that  the  proposed  work  will  have  market  potential  both  for  the  Federal  Government 
and  the  private  sector.  Currently,  there  is  an  explosion  in  the  development  of  multimedia 
technologies.  However,  the  development  costs  for  such  technologies  are  great  due  to  the  richness 
of  the  modes  of  presenting  information.  We  believe  that  not  all  training  tasks  require  full 
multimedia  technology  to  be  effective.  However,  trainers  do  not  have  a  methodology  for  making 
tradeoffs  of  available  technologies  against  resources  (both  training  time  and  money  to  procure). 
We  believe  a  tool  that  assists  in  making  these  decisions  would  have  commercial  potential. 
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Building  11,  Suite  7 
2240  B  Street 

Wright  Patterson  AFB,  OH  45433-7111 


Science  and  Technology  Division 
Engineering  Training  Systems  Competency 
Naval  Air  Warfare  Center 
Training  Systems  Division 
TSD  4.9.6 

12350  Research  Parkway 
Orlando,  FL  32826-3224 

Instructional  Systems  Competency 
Naval  Air  Warfare  Center 
Training  Systems  Division 
TSD  4.9.7 

12350  Research  Parkway 
Orlando,  FL  32826-3224 

Mr.  John  J.  Crane  Science  and  Technology  Division 

Chief  of  Naval  Education  &  Training 

NAWCTSD  Liaison  Office,  Code  1 . 0C 

Building  628,  Room  2-44 

NAS,  Pensacola,  FL  32508-5100 

Commanding  Officer 
NETPMSA 

Code  04,  ATTN:  Dr.  Nancy  Perry 
NAS,  Pensacola,  FL  32509-5100 

Department  of  the  Navy 

Office  of  the  Chief  of  Naval  Operations 
BUPERS  00H 

Washington,  DC  20350-2000 
Department  of  the  Navy 

Office  of  the  Chief  of  Naval  Operations 
0P-911H 

ATTN:  Dr.  Bart  Kuhn 

Washington,  DC  20350-2000 

Department  of  the  Navy 

Office  of  the  Chief  of  Naval  Operations 
OP-01B2E1 

Washington,  DC  20350-2000 

Office  of  Naval  Research 
Code  342CS ,  ATTN:  Dr.  Terry  Allard 
800  North  Quincy  Street 
Arlington,  VA  22217-5000 

Armstrong  Labs/Aircrew  Training  Research 

ATTN:  Dr.  Dee  Andrews 

6001  S.  Power  Road,  Bldg  558 

Mesa,  AZ  85209-0904 
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American  Psychological  Association 
Psych.  Info.  Document  Control  Unit 
750  First  St  NE 
Washington, DC  20002 

Defense  Technical  Information  Center 
8725  John  J.  Kingman  Road,  Suite  0944 
Ft  Belvoir ,  VA  22060-6218 

Commander 

Marine  Corps  Systems  Command 
<  PMTRASYS,  Code  SST 
}  Quantico,  VA  22134-5010 

^  Institute  for  Defense  Analyses 
Science  and  Technology  Division 
ATTN:  Dr.  Dexter  Fletcher 
1801  Beauregard  Street 
Arlington,  VA  22311 

National  Defense  Institute 
Research  Directorate 
Ft  McNair,  DC  20319 

OUSDR&E  (E&LS) 

The  Pentagon 

Washington,  DC  20301-3080 

DASD  (FM&P/Tng  Policy) 

ATTN:  Mr.  Gary  Boycan 

The  Pentagon,  Room  3B930 
Washington,  DC  20301-4000 

Dr.  Allen  R.  Zeman 
Special  Assistant  (Training) 

Assistant  Secretary  of  the  Navy 
(Manpoer  and  Reserve  Affairs) 

The  Pentagon 
Room  4E789 

Washington  DC  20350-1000 
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